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ABSTRACT 

This study addresses four questions: (1) What types 
of performance appraisal policy for educational staff have been 
adopted by Ontario school boards? (2) To what extent have these 
policies been implemented in practice? (3) What types of appraisal 
policies are most effective? and (4) What processes have school 
boards used to develop and implement their appraisal policies? To 
answer these questions, policies for the appraisal of teachers, 
principals, superintendents, and directors were requested from all 
126 public boards of education or county and district Roman Catholic 
separate school boards in Ontario. Based on the responses, typologies 
were deve.loped for appraisal practices for each of the roles under 
study, ami questionnaires based on these typologies were administered 
to teachers, principals, superintendents, trustees, and directors in 
a stratified sample of 30 representative school boards. Percentage 
distributions were prepared for all characteristics included in the 
typologies of performance appraisal policy, revealing which practices 
were most widespread, and the characteristics of appraisal systems 
were correlated urith their effects on the attitudes and behavior of 
those evaluated. Finally, individual case study reports were used to 
prepare a comparative analysis. Findings, reported in detail, suggest 
that formal appraisal systems can be effective, but that more effort 
must be made in imj;>lementing policies, and that stronger linkages are 
needed between performance appraisal and school boards' goals, 
organization, and job descriptions. (TE) 
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the attitudes and behaviour of those evaluated. Finally, individual case study reports '/ere 
prepared, and a compiirati ve analysis of these written. 

The typologies used for all roles included five major cateaories: preparr^tion for 
evaluation, data collection, reporting and follow-up evolution of policy, and impact of 
policies and practice. 



Findings of importance include the following: 



1. As of November 1982, 73 per cent of all school boards in Ontario had adopted policies for 
teacher evaluation; 53 per cent for principals, 42 per cent for superintendents; and 36 per 
cent for directors. 

2. Policies tended to become less detailed for positions higher in the hierarchy; 
objectives-based appraisals, though occurring at all levels, were more frequently used for 
positions higher in the hierarchy. 

3. The primary purpose of appraisal at all levels was developmental; appraisal for 
administrative purposes was somewhat more com^Tion at the director's level. 

4. Evaluation by a single individual is most common at a'l levels, except that of director, 
though information was often collected from several sources. Specifying the length of time 
required to collect information was common for teachers and principals, but rare for 
Superintendents and directors. 

5. Requirements for written reports and plans to follow up on appraisal are most common for 
teachers, less common for principals and superintendents, and least common for directors. 

6. Frr the most part, practices followed policies quite closely; where differences were 
reported, they often reflected a failure to maintain proper schedules, to do required numbers 
of observations, and the like. Those who were eva.jated were more likely to report such 
discrepancies than were those who conducted the evaluations. 
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7. Typically, evaluation was reported to increase the morale of staff members and to provide 
them .^ith a few ideas on how to improve. The most impact was reported , superintendents and 
the least by teachers. Directors were the onlv group whose morale was not increased by 
appraisal . 

8. Characteristics of appraisal systems associated with more effective teacher or principal 
appraisals included the holding of pre-evaluation conferences, longer pre-conferences , clear 
communication of purposes of the appraisal, the use of general and specific notes, reports 
under several headings, post-evaluation conferences, and the makirig of plans. Characteristics 
associated with less effective appraisals included the use of student absenteeism data and 
appraisals that began with informal visits from the evaluator. 

9. Appraisal of administrators, especially of principals and superintendents, is seen as 
being more effective than that of teachers, even though policies for the latter are more 
detailed. The burden of appraising large numbers of teachers may make the relative effort 
devoted to each teacher's appraisal much less than that devoted to each administrator's 
appraisal . 

10. Most respondents would prefer more sources of data be used in their appraisal; use of 
self-evaluation questionnaires was approved by all groups, but rarely used in practice. 

11. Appraisal criteria related to out-of-classroom activities were not widely supported among 
teachers, though separate school and francophone teachers were more supportive of their use 
than were teachers in public boards of education. A similar situation with regard to 
out-of-school activities applies to principals. Superintendents and directors believe more 
emphasis in their own appraisal should be pl^.ced on knowledge of laws and regulations; 
directors believe more emphasis should be placed on their knowledge of schools and programs. 

12. Placing staff under review, dismissal of staff, and the filing of appeals, grievances, 
and lawsuits by staff were least common in school boards where appraisal systems were reported 
to be most effective. 
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Several conclusions are drawn from the study. First, formal appraisal systems clearly 
can be effective, they need not be nollow rituals, as some have suggested. Second, more 
effort must be made in implementing policies; both evaluators and evaluatees need to know how 
to use evaluation processes effectively. Third, stronger linkages are needed between 
performance appraisal and school boards' goals and objectives, their organizational structure, 
and the job descriptions for individual positions. Results of appraisal are too rarely used 
to improve school board organization to prevent recurring problems that become evident in 
appraisal, such as the lack of consensus over the role of the teacher in activities outside 
the classroom. To accomplish this end, formal reviews of appraisal practices and their 
effectiveness are needed. 
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CHAPTER 1 
INTRODUCTION 



Purpose of Study 



The purpose of this report is to describe, in considerable detail, the findings of a major 
study of current performance appraisal policies and practices in Ontario school boards. The 
study, conducted under contract with the Ontario Ministry of Education, sought to answer four 
questions: What types of performance appraisal policy for educational staff have been adopted 
by Ontario school boards? To what extent have these policies been implemented in practice? 
What types of appraisal practices are most effective? And, what processes have school boards 
used to develop and implement their appraisal policies? 

To answer these questions, the scope of the research called for in the contract set out 
six objectives: 



To collect, from all Ontario school boards, existing policies, procedures, and 
instruments for the performance appraisal of teachers, principals, and supervisory 
officers, including the chief executive officer; 

To develop a typology of appraisal systems and prepare a report describing their 
distribution among Ontario school boards; 

To survey teachers, principals, supervisory officers, and trustees in a 
representative sample of boards to determine the extent of implementation of the 
appraisal system in each board, its perceived level of effectiveness, and the 
staff's satisfaction with it; 

To prepare a statistical report describing and analysing survey results; 
To select, from the sample of boards, eight and to conduct case studies in them; 
To prepare and submit to the Minister two final reports, one technical and one 
non-technical, describing tiiese studies. 



The investigation was to be thorough, with attention paid to public and separate boards, 
boards in all regions in Ontario, and boards in which French was the language of a large 
proportion of the population. 




Design of Study 



Since detailed descriptions of each phase of the project are provided in subsequent chapters, 
only an overview of the study's strategies is presented here. 

For the most part, this was treated as an exploratory study in which each stage was built 
on the previous stages. First, existing policies on the performance appraisal of teachers, 
principals, superintendents, and directors were requested from all Ontario school boards. As 
well, additional information needed to select a sample of school boards was collected on a 
brief screening questionnaire. Second, the appraisal policies collected were analysed to 
develop a typology, the framework for which is sketched below. Third, a description of the 
frequency of different types of policy element was prepared. Fourth, questionnaires were 
developed based upon the typology; these were sent to school boards in diverse settings with 
diverse kinds of appraisal policy. Fifth, statistical data from the survey were analysed. 
Sixth, eight school boards were selected for case studies which probed the developmental 
processes and the like that could not be adequately captured in a questionnaire survey. 
Finally, reports were prepared describing the results. 

The study took place over a 23-month period from October 1982 to August 1984. Policies 
were collected in late 1982; the Ergl ish-language questionnaire survey was conducted in 
spring 1983; seven case studies and the French- language questionnaire survey were conducted in 
fall 1983; the remaining case study, in a board with a large proportion of francophone 
students, was carried out in winter 1984. The final reports were prepared during the spring 
and summer of 1984. 

Framework for Study 

To ensure an organized approach to the topic of this study, we developed a 'ramework or model, 
displayed in figure 1, that maps out the organizational structure and processes involved in 
performance aporaisal systems. The framework should be looked upon as a device to organize 
materials and ideas, not a prescriptive model for action. 



2 

ERIC 



Figure 1 

Systems Model for Performance Appraisal 
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The left half of the model describes the organizational structure of a school board. 
Beginning with goals and objectives, a school board creates schools, departments, and units of 
various types, each with its own responsibilities, so that the overall goals can be achieved. 
Ar. important component of any unit is the job descriptions that are developed for positions 
within that unit. In the case of schools, the job descriptions for teachers and principals 
are clearly the most central, but, with the increased size and complexity of educational 
systems, there are numerous jobs sufficiently unique to warrant their own description. 

With collective bargaining, the nature of the employment contract has become of 
increasing importance; debates about changes in the role of staff, particularly teachers, are 
often resolved at the bargaining table. Individuals, with their own motivations, work within 
the constraints imposed (or ...portunities created) by their job descriptions and contracts. 
The quality of their performance on the job is a major determinant of the success with which 
the system as a whole achieves its objectives. This brings us to the topic of performance 
apprai sal . 

If all individuals always performed at their best, if in hiring peo,jle one could always 
be correct in one's judgement, if the abilities of individuals never declined, if new demands 
were never placed on employees, then one would have no need for staff appraisal procedures. 
Given these conditions are never met, some type of evaluation of employees is necessary. 

The major thrust in recent decades, as far as staff evaluation is concerned, has been the 
need to improve day-to-day supervision. Formal systems of evaluation, especially that used 
for teachers, fell into disrepute. Such formal systems, often involving brief observations of 
a teacher in the classroom, were condemned as hollow rituals which were at best unproductive 
and at worst destructive of morale and creativity. Nevertheless, concerns about the quality 
of education - and indeed of the quality of work in many public and private enterprises 
reawakened interest in formal systems of staff evaluation. It is wHh such formal systems 
that this study is concerned. 

Turning now to the right side of figure 1, we can see that both organizational goals and 
structure determine the purposes of performance appraisal. These purposes, along with job 
descriptions and employment contracts, then suggest the criteria for appraisal; the criteria, 
along with employment contracts, condition the sources, types, and methods of data collection 
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concerning an individual's performance; and finally, the analysis of those analyses, 
employing a set of standards to compare expected performance with actual performance, results 
in decisions that are shared with the evaluatee and used, perhaps, to plan a set of follow-up 
activities. Subsequently, the results of one or more appraisals may affect all aspects of the 
organization (job performance, individual objectives, contracts, job descriptions, and 
organizational structure) and the appraisal process itself (data collection, criteria, and 
purposes). 

In practice, some broader classifications of the stages in the appraisal process proved 
useful. Five categories — preparation for appraisal, data collection, reporting and 
follow-up, evolution of policy, and impact of policy and practice — are labeled on figure 1. 
These five are used repeatedly in the subsequent chapters to organize the reporting of 
findings and discussion. Though they are developed in detail in chapter 2, the meanings of 
these categories are summarized here. 

Preparation for performance appraisal includes four major aspects: planning, purposes, 
criteria, and standards. Planning includes such matters as notification of the evaluatee and 
the holding of pre-evaluation conferences. Purposes include the intended outcomes (e.g., 
improvement of instruction) as well as possible decisions (e.g., whether or not a permanent 
contract should be granted). Criteria, as used here, refer to indicators that measure 
qualities or behaviours; some might be quite complex, such as the style of classroom 
management, while others might be straightforward, such as punctuality. Standards, then, 
relate to the extent to which expectations regarding criteria are fulfilled. (In the 
dictionary definition, criteria and standards are synonyms; the distinction we have made, 
however, is both useful and common as far as performance appraisal is concerned.) 

Data collection, as suggested in figure 1, includes sources and types of information. 
Method of collection is refined into two factors — who collects the information and the time 
spent in the process. 

Reporting and follow-up include the nature of the report that is provided, its 
destination, with whom it is shared, and any foliow-up activities, such as plans for action, 
that are developed. 
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Evolution of policy includes the process by which it is developed (including who 
participated), the activities undertaken to implement it, reviews of the policy, and the 
extent of specificity found within the policy. One important distinction in this regard is 
between the unitary policies and those that separate administrative from developmental 
purposes. 

Finally, the impact of policy and practice includes the degree of compliance with the 
policy, the extent of effort expended in its implementation and administration, and the nature 
and degree of its impact. 



Overview 



This introductory chapter is primarily concerned with the framework used in conducting the 
study and in reporting the results. The next three chapters relate directly to the points 
outlined in the scope of the project at the beginning of this chapter. In particular, chapter 
2 describes the results of the screening survey, the analysis of appraisal policies, and the 
typology called for in points 1 and 2 of the scope of toe research. Data from the screening 
questionnaire indicate the prevalence of performance appraisal policies in late 1982, when 
these policies had been adopted, and, in the views of the directors who completed the 
questionnaires, how well implemented and effective these were. The analysis of policies 
includes four sections, one each devoted to the appraisal of teachers, principals, 
superintendents, and directors. A typology is developed for each, using four of our major 
categories — preparation, data collection, reporting and follow-up, and evolution of policy. 
Within each major category, subcategories are developed, and the characteristics of appraisal 
policies are placed within these. For each characteristic, frequency counts are reported to 
indicate the prevalence of its use. To illustrate different practices, numerous excerpts from 
policies are included. The chapter concludes with a comparison of the appraisal policies for 
teachers, principals, superintendents, and directors. 

Chapter 3 describes the results of the survey called for in points 3 and 4 of the scope 
of the research. The chapter begins with a detailed exposition of the <;ample design and 
survey methods. This is followed by a description of the actual performance appraisal 
practices experienced by the four groups surveyed. As well, respondents' views as to how they 
and their colleagues ought ideally to be formally evaluated are reported. This part of the 
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chapter is organized in the same manner as is the body of the preceding chapter, with separ=»te 
sections for each rcle and subsections for the various categories used in the typologies 
developed in chapter 2. For the most part overall statistics are reported, with differences 
noted for those in different types of school (public elementary, public secondary, or 
separate), different types of school board (public or separate), and schools with different 
languages of instruction (French or English). Also provided are an analysis of the 
relationships of the characteristics of performance appraisal practices to the effectiveness 
of the evaluation process for teachers and principals; a demographic profile of the 
questionnaire respondents; and an analysis of the correlates of the numbers of teachers who 
are placed under review, dismissed, and file an appeal, grievance, or lawsuit. The chapter 
concludes with a comparison of the findings for those in differant roles. 

Chapter 4 reports a cross-case analysis of the eight case studies. After a description 
of the purpose and methodology of these studies, the chapter compares each phase of the 
appraisal process for directors, superintendents, principals, teachers, and consultants in 
each of the eight school boards studied. While other chapters emphasize the typical or 
overall situation in Ontario school boards with regard to performance appraisal, this chapter 
emphasizes variation among boards, linking the unique situation in a board to its approach to 
staff evaluation. The chapter concludes with a synthesis of the results for the various roles 
and an assessment of where major difficulties lie. 

The fifth and final chapter draws conclusions from the findings and suggests possible 
implications for practice. 



This volume and its two companion volumes of appendices comprise the technical report 
called for in point 6 of the scope of the research. Volume 2, Appendices A and B, includes 
statistical tables and copies of the survey instruments on which chapter 3 of this report is 
based; Volume 3, Appendix C, includes copies of the eight case studies on which chapter 4 is 
based. This report is supplemented by a non-technical report, which provides a precis of the 
itudy and its implications. 
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CHAPTER 2 

PERFORMANCE APPRAISAL POLICIES IN ONTARIO SCHOOL BOARDS 



School beards in Ontario have developed, as one strategy for providing quality education, a 
considercible range of ways of dealing with performance appraisal of certificated education 
staff. Some jurisdictions have very detailed and specific procedures, while others rely upon 
administrative practices, often undocumented, developed by individual supervisors in the 
course of their duty. 

Both those responsible for making decisions regarding performance appraisal and those 
most affected by such appraisal need comprehensive, accurate, and accessible information about 
the current range of policies and practices in Ontario. 

Accordin.^ly, the first phase of this project was designed to provide an information base 
regarding the ex.ent, distribution, diversity, and perceived effectiveness of existing 
approaches to the appraisal of certificated education staff in Ontario boards. To achieve 
this objective, a screening questionnaire was sent to all school boards in Ontario, along with 
a request for copies of existing policies, procedures, and instruments for the appraisal of 
teachers, principals, and supervisory officers, including directors. Subsequently, this 
information was used (1) in the design of a detailed questionnaire to be sent, in the second 
project phase, to staff in 30 boards, and (2) to select the sample of school boards to receive 
these questionnaires. 

This chapter provides (1) a description of the results of the screening survey sent to 
all school boards, (2) a typology developed to analyse and describe various systems of 
performance appraisal, and (3) the content and prevalence of different types of evaluation 
policies at all levels. 

Methods of Data Collection and Aalysis 

In October and November of 1982, directors of all 187 Ontario school boards were sent letters 
requesting policies, guidelines, and instruments relevant to their board's performance 
appraisal practices. Where the appraisal system was not adequately described in existing 
documents, respondents were asked to submit a letter describing whatever system was in use. 
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Accompanying the letters to directors were an abstract describing the overall research project 
and a one-page screening questionnaire (figure 2), which asked for student enrolment, as well 
as the year of adoption of their system, extent of its implementation, and perceived 
effectiveness of the system for educational staff and others in their school and board. 



9 2b 



Figure 2: Screening Questionnaire 

THE ONTARIO INSTITUTE FOR STUDIES IN EDUCATION 

PERFORMANCE APPRAISAL STUDY 

STUDY OF SCHOOL BOARD CHARACTERISTICS 

1. HimtQfBoird: Code: 

2. tnrolirtnl (Septtmbtr 1982): ! 

3 In whit yeir did your board adopt Iht ptrfomanct ippraisil lysiefn dtjcri 
In Iht docuffltnli or Ittttr you art fon^rding to ui7 

Year Mot ^ppUcablt 

i. for teachtri? 19 

b. for pr'ncipali? 19_ 

C. for iuptrintendtnt',f 19 

6, for tht dirtctor? 19 

t. for others? 19 

D'wninti: 



Do you btlleve the appraisal systtie has been loiplt-enttd? ("tase clrclt your response.) 

As Approximately Very Differently Hot 
Described. As Described from 0#ser1bed_ Applicable 

a. for teachers 3 2 10 

b. for principals 3 2 10 

c. for superintendents 3 2 10 

d. for the director 3 2 10 

e. others 3 2 | 0 

Comments: 



Kdm effectively do you believe your perfonnance appraisal system 1$ achieving Its 
objectives? (Please circle your response.) 

Very Somewhat Mot Mot 

Effectively Effectively Effectively Applicable 



a. 



for teachers? 3 



d. for the director 



2 1 0 



b. for principals 3 2^0 

c. for superfntsndtntj? 3 2 10 



3 2 10 



e. far others? 3 2 10 

f. for schools? 3 2 10 

9. for the system as in 
a sihole? 3 J? > 0 

ComnentS! — 



<. Perfomance appraisal policies, guidelines and Instru-^ents 

(— are being sent under 

a. I 1 are enclosed. b. | | separate cover. 

Thank you for your assistance. 



PicoiC fittuXn by HOvVftbtA 19, 1912 

Performance Anaralsal Study 

Stephen 6. !av^;on. Principal Investigator 

Departnent Educational Administration 

The Ontario Institute for Studies In Education 

2S2 lloor Street West 

Toronto, Ontario H5S 1V6 
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As is described in a subsequent section on sample design, initial rates of return from 
boards other than public boards of education or county and district Roman Catholic separate 
school boards were very low. Consequently, we dpcided, with the approval of the project's 
Advisory Group, to report only on the 77 Dub^•: and 49 Roman CathoUc separate school boards 
in the province. 

Those school Doards that did not reply were asked again, by letter in December 1982 and 
by telephone in ear.y January 1983, to submit documents and screen^'ng questionnaires to the 
project team. 

Screening questionnaires and/or replies regarding appraisal practices (i.e., relevant 
documents, a letter describing practices, or letters indicating an absence of any appraisal 
system) were received from 75 public (97 per cent) and 44 separate school boards (90 per 
cent). 

Analysis of Screening Questionnaire 

In addition to the information collected from the screening questionnaire, additional 
information concerning all school boards was obtained from other sources. 

The proportion of French sc:.cols was obtained by dividing the number of French schools by 
the total number of schools for each board, using school listings from the Ontario Ministry of 
Education's 1982/1983 Directory of Education . 

The extent to which enrolment was or was not declining was calculated from the 1982 
figures obtained in the screening survey (or, where there was no reply, from thc- 
Dirfectory of Education ) and enrolment figures obtained in the Ministry's S urvey of School 
Board 1977 Estimates (or, for a small number of boards for which information was not included 
in this survey, enrolment as of September 1977 reported in the Ministry's Public School 
^"''"^"'^"^ and Secondary School Enrolment). Specifically, the 1982 enrolments were subtracted 
from the 1977 enrolments, and the difference divided by the 1977 enrolment figure. The 
five-year period was selected to assess the dominant enrolment trend in a board and to 
eliminate minor year-to-year variations. 
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Frequency distr ibutTons for all variables were calculated for all 126 boards, and for 
public and separate boards separately. These distributions were used both in the selection of 
the sample for the major survey and in the analysis of policies. 

Analysis of Policies 

For the 119 boards which provided information about appraisal policies, guidelines, 
instruments, and/or prescribed practices (all of which are referred to as "policy" here), 
information related to the appraisal of teachers, principals, superintendents, and directors 
was coded using a number of categories concerned with prescribed or suggested practices, 
including (1) preparation for appraisal (purposes, criteria, and standards), (2) data 
collection (sources of information, data collection, method of collection, eval uator(s) , time 
spent). (3) reporting and follow-up, (4) differentiation among procedures for certain 
circumstances and categories of personnel, and (5) evolution of policy (development, 
implementation, and review). The precise usage of the various categories is elaborated upon 
during the discussion of results. 

For each role and each board, the presence or absence of various components in the policy 
was determined. With respect to a few categories, the degree to which the component applied 
was recorded. Subsequently, the relative frequency with which the various policy components 
occurred was calculated. 

For boards appraising teachers and/or principals, cluster analyses were conducted to 
produce groupings of similar types of system. To do this, for teacher and principal policies 
separately, responding boards were ranked according to the number of components included in 
their policy, and components were ranked according to the number of boards reporting their 
inclusion. A matrix of rank-ordered boards by rank-ordered components was prepared and 
examined for clusters. For teachers, this technique was used twice, first using 30 
components, and then using the 11 components which seemed particularly likely to reflect basic 
differences in underlying assumptions among policies. For principals, the technique was used 
only once, with 18 components. 
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Results 



This section begins with an overview of the results of the screening questionnaire. 
Subsequently, analysis of the various performance appraisal policies are presented for 
teachers, principals, superintendents, and directors. 

Screening Questionnaire 

In the following tables giving results for the screening questionnaire, percentages refer to 
the entire population of 77 public and 49 separate schools. In fact, only 73 public and 43 
separate actually returned the screening questionnaire. Therefore, for most items the 
percentage of non-responses is stated so that total equals 100 per cent (the NA response 
includes those who circled "not applicable" and cases where "no answer" was received). 

Although questionnaires were sent to the director of each school board and the majority 
of them were completed by the director, in a small number of boards, directors referred the 
questionnaire to an appropriate superintendent for completion. 

Table 1 shows the regional distribution of Ontario public and separate boards included in 
the study. The column total gives the disf-ibution of school boards by region and the row 
total gives the distribution of boards by type. 



Distribution 


of Ontario 


School Boards 


by Region and 


bv Tvoe 


Region 


Public 


Separate 


Total 






n 


% 


n 


% 


n 


% 


Northwest 


9 


11 .7 


6 


12.2 


15 


11 .9 


Midnorthern 


9 


11 .7 


5 


10.2 


14 


11 .1 


Northeastern 


10 


13.0 


7 


14.3 


17 


13.5 


Western 


12 


15.6 


9 


18.4 


21 


16.7 


Central 


28 


36.4 


15 


30.6 


43 


34.1 


Eastern 


9 


1 1 .7 


7 


14.3 


16 


12.7 


Total 


77 


61 .1 


49 


38.9 




100.0 


s 2 through 


6 indicate 


respectively, 


the years in which trie 



appraisal systems for teachers, principals, superintendents, directors, and other catrgories 
of staff, by type of board. 
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T.blP 2: Distribution of Boards b y Year of Adoption of rnrrPnt A ppra^'sal System for Teachers 
and by Type 

Y,,r Public Separate 

(n.77) (n-49) (n-126) 





n 


A* 


n 


i 


n 


% 




1 


1 .3 


0 


0.0 


1 


0.8 




2 


2.6 


1 


2.0 


3 


2.4 




1 


1.3 


0 


0*0 


1 


0.8 




3 


3.9 


1 


2.0 


4 


3.2 




4 


5.2 


3 


6.1 


7 


5.6 


1 077 


1 


1.3 


2 


4.1 


3 


2.4 




3 


3-9 


0 


0.0 


3 


2.4 


1979 


6 


7.8 


4 


8.2 


10 


7.9 


1980 


8 


10.4 


4 


8.2 


12 


9.5 


1981 


9 


11.7 


4 


8.2 


13 


10.3 


1982 


6 


7.8 


4 


8.2 


10 


7.9 


NA 


33 


42.9 


26 


53.1 


59 


46.8 
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Table 3; Distribution of Boards by Year of Adoption of Current Appraisal System for 
Principals and by Type 



Year Public Separate Total 

(n-77) (n«49) (n=126) 





n 


/O 


n 


% 


n 




1970 


1 


1.3 


0 


0.0 


1 


0.8 


1972 


2 


2.6 


1 


2.0 


3 


2.4 


1973 


1 


1-3 


0 


0.0 


1 


0.8 


1975 


5 


3.9 


1 


2.0 


4 


3.2 


1976 


4 


5.2 


3 


6.1 


7 


5.6 


1977 


1 


1.3 


2 


4.1 


3 


2.4 


1978 


5 


3.9 


0 


0.0 


3 


2.4 


1979 


6 


7.8 


4 


8.2 


10 


7.9 


1980 


8 


10.4 


4 


8.2 


12 


9.5 


1981 


9 


11.7 


4 


8.2 


13 


10.3 


1982 


6 


7.B 


4 


8.2 


10 


7.9 


NA 


33 


42.9 


26 


53.1 


59 


46.8 



Table 4: Distribution of Boards by Year of Adoption of Current Appraisal System for 
Superintendents and by Type 

Year Public Separate Total 

(n-77) (n-49) (n=1?b' 





n 




n 


% 


n 


•f 


1973 


2 


2.6 


0 


0.0 


2 


1 .6 


1974 


1 


1.3 


0 


0.0 


1 


O.c 


1975 


1 


1.3 


1 


2.0 


2 


1 .6 


1976 


3 


3.9 


1 


2.0 


4 


3.2 


1977 


1 


1.3 


0 


0.0 


1 


0.8 


1978 


2 


2.6 


0 


0.0 


2 


1 .6 


1979 


6 


7.8 


3 


6.1 


9 


7.1 


1980 


10 


13.0 


3 


6.1 


13 


10.3 


1981 


8 


10.4 


1 


2.0 


9 


7.'' 


1982 


7 


9.1 


3 


6.1 


10 


7.9 


NA 


36 


46.8 


37 


75.6 


73 


57.9 
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Table 5; Distribution of Boards by Year of Adoption of Current Appraisal System for 
Directors and by Type 



Year Public Separate Total 

(n-77) (n-49) (n-126) 





n 


<* 


n 




• n 


% 


1973 


2 


2.6 


0 


0.0 


2 


1 .6 


1975 


1 


1.3 


1 


2.0 


2 


1.6 


1976 


2 


2.6 


1 


2.0 


3 


2.4 


1977 


1 


1.3 


1 


2.0 


2 


1 .6 


1978 


3 


3.9 


0 


0.0 


3 


2.4 


1979 


6 


7.8 


2 


4.1 


e 


6.3 


1960 


10 


13.0 


3 


6.1 


13 


10.3 


1981 


4 


5.2 


2 


4.1 


6 


4.8 


1962 


3 


i.S 


3 


6.1 


6 


4.6 


NA 


45 


58.5 


36 


73.5 


61 


64.3 



Table 6: Distribution of Boards by Year of Adoption of Current Appraisal System for Others 
and by Type 



Year Public Separate Total 

(n-77) (n-49) (n-126) 





n 


% 


n 


% 


n 


% 


1972 


1 


1.3 


0 


0.0 


1 


0.8 


1973 


1 


1.3 


0 


0.0 


1 


0.8 


1975 


2 


2.6 


1 


2.0 


3 


2.4 


1976 


1 


1.3 


0 


0.0 


1 


0.8 


1978 


0 


0.0 


1 


2.0 


1 


0.8 


1979 


1 


1.3 


1 


2.0 


2 


1 .6 


1980 


9 


11.7 


1 


2.0 


10 


7.9 


1981 


t 


5.2 


0 


0.0 


4 


3.2 


1982 


2 


2.6 


4 


8.2 


6 


4.8 


NA 


56 


72.8 


41 


83.6 


97 


77-0 



3j 



16 



Examination of the total number of NAs in the five tables shows that more boards have 
appraisal systems for teachers than for, in order, principals, superintendents, directors, or 
other staff categories. 



That the percentages of boards adopting are higher in the last four years than in the 
earlier years should not in itselT je taken as evidence that boards have waited until recently 
to adopt systems. Respondents were asked to indicate the year of adoption of the system 
described in the documents they were submitting (i.e., their current system), and many of 
these systems may have been modifications or replacements of previously adopted systems. 

There is a tendency for separate school boards to have a higher NA percentage than public 
boards in all categories; this difference is greatest with respect to appraisal systems for 
superintendents and least with respect to those for teachers. 

Tables 7 to 11 indicate the degree to which boards perceive the current appraisal systems 
to be implemented. 

Table 7: Degree of Implementation of Teacher Appraisal System by Type of Board 



Degree of 
Implementation 


Fab 


xic 
77) 


Separate 

(n-49) 


Total 
(n-126) 




n 


% 


n 


% 


n 


% 


As described 


29 


37.7 


16 


32.7 


45 


35.7 


Approximately 
as described 


21 


27.3 


14 


28.6 


35 


27.8 


Very differently 
from '^.'iscribed 


0 


0.0 


1 


2.0 


1 


0.8 


NA 


27 


35.1 


18 


36.7 


45 


35.7 
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Table 8: Degree of Implementation of Principal Appraisal System by Typ e of Board 



Degree of Public Separate Total 

Implementation (n-77) (n-49) (n-126) 





n 


% 


n 


JO 


n 




As described 


24 


31.2 


13 


26.5 


' 37 


29.4 


Approximately 
as described 


20 


26.0 


9 


13.4 


29 


23.0 


Very differently 
from described 


1 


1.3 


1 


2.0 


2 


1 .6 


NA 


32 


41 .6 


26 


53.0 


58 


46.1 



Table 9: Degree of Implementation of Superintendent Appraisal System by Type of Board 



Degree of 
Implementation 


Public 
{n-77) 




Separate 
{n-49) 


Total 
(n-126) 




n 




n 




n 


% 


As described 


21 


27.3 


6 


12.2 


27 


21.4 


Approximately 
as described 


17 


22.1 


6 


12.2 


23 


18.3 


Very differently 
from described 


1 


1.3 


0 


0.0 


1 


0.8 


NA 


38 


49.4 


37 


75.5 


75 


59.5 



Table 10: Degree of Implementation of Director Appraisal System by Type of Board 
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Degree of 
Implementation 


Public 
(n-77) 




Separate 
(n-49) 


Total 
(n-126) 




n 




n 


% 


n 


% 


As described 


18 


25.4 


6 


12.2 


24 


19.0 


Approximately 
as described 


11 


14.3 


5 


10.2 


16 


12.7 


Very differently 
fror described 


2 


2.6 


0 


0.0 


2 


1.6 


NA 


46 


59.8 


38 

18 


77.6 

35 


84 


66.7 



Table 11: Degree of Implementation of Other Appraisal Systems by Type of Board 



Degree of Public Separate To .1 

Implementation (n-77) (n«49) (n«l 26) 

n % n % n % 



A3 described 8 10.4 3 6.1 11 8.7 
Approximately 

as described 10 13.0 5 io.2 15 11.9 
Very differently 

from described 0 0.0 0 0.0 0 0.0 

NA 59 76.6 41 83.6 100 79.3 



In looking through tables 7 to 11, we see, as in tables 2 to 6, an increasing percentage 
of boards in the NA category. 

Among those boards which did provide an estimate of degree of implementation (e.g., 42 
boards for directors), the percentage replying that the system was implemented as described 
was 53 for superintendents, 54 for principals, 56 for teachers, 59 for directors, and only 42 
for other staff categories. The patterns for public and separate boards considered separately 
are very similar. 

Tables 12 to 16 indicate the degree to which the current systems are considered to be 
effective in achieving their objectives. Tables 17 ano 18 present similar findings for 
success at achieving school and school system objectives. 

Table 12: Effectiveness of Teacher Appraisal Systems in Achieving Their Obj ectives by Type 
of Board — 



Degree of Public Separate Total 

Effectiveness (n-77) (n-49) .(n-126) 





n 


% 


n 


% 


n 


% 


Veiy effective 


20 


26.0 


9 


18.4 


29 


' 23.0 


Somewhat effective 


28 


36.4 


16 


32.7 


44 


34.9 


Not effective 


2 


2.6 


4 


8.2 


6 


4.8 


NA 


27 


35.1 


20 


40.8 


47 


37.3 
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Table 13: Effectiveness of Principal Appraisal Systems in Achieving Their O bjectives by 
T ype of Board 



Degree of Public Separate Total 

Effectiveness (n-77) (n-49) (n.126) 





n 


% 


n 


% 


n 


<* 


Very effective 


17 


22.1 


5 


10,2 


22 


17.5 


Somewhat effective 


23 


29.9 


16 


32.7 


39 


31.0 


Not effective 


2 


2.6 


1 


2.0 


3 


2.4 


NA 


35 


45.5 


27 


55.1 


62 


49.2 



Table 14: Effectiveness of Superintendent Appraisal Systems in Achieving Their O bjectives 
by Type of Board 



Degree of Public Separate Total 

Effectiveness (n-77) (n«49) (n-126) 





n 


% 


n 


% 


n 


% 


Very effective 


13 


16.9 


3 


6.1 


16 


12.7 


Somewhat effective 


22 


28.6 


11 


22.4 


33 


26.2 


Not effective 


2 


2.6 


0 


0.0 


2 


1.6 


NA 


40 


52.0 


35 


71.4 


75 


59.5 



Table 15: Effectiveness of Director Appraisal Systems in Achievinr T heir Objectives by Type 
of Board 



Degree of Public Separate Total 

Effectiveness (n-77) (n-49) (n-126) 





n 


% 


n 


% 


n 


% 


Very effective 


12 


15.6 


2 


4.1. 


14 


11.1 


Somewhat effective 


14 


18.2 


11 


22.4 


25 


19.8 


Not effective 


3 


3.9 


1 


2.0 


4 


3.2 


NA 


48 


62.4 


35 


71.4 


83 


65.9 
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l^^^^ — Effectiveness of Appr aisal Systems for Others in Achievi ng Their Obiectives by 
Type of Board ^ ^ 



Degree of Public Separate Total 

Effectiveness (n-77) (n.49) (n-126) 





n 


% 


n 




n 


% 


Very effective 


9 


11.7 


2 


4.1 


1 1 


6.7 


Somewhat effective 


13 


16. 9 


9 


16.4 


22 


17.5 


Not effective 


2 


2.6 


0.0 


0.0 


2 


1.6 


NA 


53 


66.9 


38 


77.6 


91 


72.2 



^^^^^ Effectiven ess of Appraisal Systems in Achieving School Objectives by Type of Board 

Degree of Public Separate Tnt«i 



Effectiveness 




77) 


(n»49) 




126) 




n 


% 


n 


% 


n 




Very effective 


5 


6.5 


2 


4.1 


7 


5.6 


Somewhat effective 


17 


22.1 


13 


26.5 


30 


23.8 


Not effective 


2 


2.6 


1 


2.0 


3 


2.4 


NA 


53 


68.9 


33 


67.3 


66 


66.2 



of^Board' ^^^^^^^^^"^ ^^ Appraisal Systems in Achieving School System Objectives by Type 



Degree of Public Separate Total 

Effectiveness (n-77) (n-49) (n-126) 





n 


% 


n 


% 


n 




Very effective 


7 


9.1 


2 


4.1 


9 


7.1 


Somewhat effective 


13 


16. 9 


13 


26.5 


26 


20.6 


Not effective 


1 


1.3 


1 


2.0 


2 


1.6 


NA 


56 


72.7 


33 


67.3 


89 


70.6 
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With perceived effectiveness, as with year of adoption and perceived degree of 
implementation, there is a steady increase as one mover> up the hierarchy from teacher 
director in the number of NA boards. 



Among those boards which did give estimates of the effectiveness of various appraisal 
systems in achieving their objectives (e.g., 52 boards for superintendents), the percentage 
regarding their systems as "very effective" ranged from 31 for superintendents and for others, 
through 33 for directors, 34 for principals, and 37 for teachers. For effectiveness of 
performance appraisal at achieving school and system objectives, the analogous percentages 
were 18 and 24 respectively. 

In this set of tables (12 to 18), there was a marked difference in the pattern of 
responses for public and separate boards. In all tables, a greater proportion of public 
boards and a smaller proportion of separate boards indicated that they regarded their systems 
ds very effective. 

Teacher Appraisal Policies 

Of the 119 boards that replied about staff appraisal, 89 provided some type of information 
about the nature of their teacher evaluation policy. In table 19, various possible components 
of a teacher policy are listed and, for each, the number of boards with policies specifying 
that feature are listed. In addition, for each component, the number of boards is shown as a 
percentage of both all 119 responding boards and the 89 boards providing ^.^eacher information. 

Table 19: Numb er and Percentage of Boards Specifying Particular Components i n Teacher 
Appraisal Policies 

Component Number of Percentage Percentage 

Boards of All 119 of 89 

Responding Boards with 

Boards Teacher 
Policies 

Preparation 

Purposes 

development/ 
• improvement: 

general 42 55.3 47.2 

development/ 
improvement : 

specific 67 56.3 75.3 
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Table 19 (continued) 



Component 



Number of 
Boards 



Percentage 
of All 119 
Responding 
Boards 



Percentage 
of 89 

Boards with 

Teacher 

Policies 



administrative : 

general 18 

administrative : 
specific 32 

routine 6 

Criterion domains 

classroom 

performance 63 

school and 
community 

involvement 47 

interpersonal 
relationships 55 

personal 

qualities 32 
professional 

development 46 

contribution 
to religious 

education 10 
Standards (explicit) 0 

Activities prior to 
appraisal 

notification in 

person 0 

notification by 
memorandum 4 

pre-conference 41 

Objective-setting 38 



15.1 

26.9 
5.0 

52.9 

39.5 
46.2 
26.9 
38.7 

8.4 
0.0 



0.0 

3.4 
34.5 
31.9 



20.2 

36.0 
6.7 

70.8 

52.8 
61 .8 
35 9 
51.7 

11.2 
0.0 



0.0 

4.5 
46.1 
42.7 
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Table 19 (continued) 



Component 



Number of 

Boards 



Percentage 
of All 119 
Responding 
Boards 



D«ta collection 

Data collection 
methods 

observation 

examination of 
documents 

interview 

other 

Sources of 
information 

teacher being 
evaluated 

principal or 
vice-principal 

teachers 

parents 

stud ents 

others 

£valuator(s) 

teacher being 
evaluated 

principal 

vice-principal 

superintendent 

Director 

department head 

coordinator 

others 



77 

11 
12 
5 



79 
1 

0 
0 
0 
5 



21 
79 
36 
45 

8 
17 

1 

4 



64.7 

9.2 
10.1 
4.2 



66.4 

0.8 
0.0 
0.0 
0.0 
4.2 



17.6 
66.4 
30.2 
37.8 
6.7 
14.2 
0.8 
3.4 



Percentage 
of 89 

Boards vith 

Teacher 

Policies 



86.5 

12.4 
13.5 
5.6 



88.8 

1.1 
0.0 
0.0 
0.0 
5.6 



23.6 
88.8 
40.4 
5U.5 
o.O 
19.1 
1.1 
4.4 
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Table 19 (continued) 



Component Number of Percentage Percentage 

Boards of All 119 of 89 

Responding Boards with 
Boards Teacher 
Policies 



Frequency: 
-for permanent 
teacher 67 56.2 75.3 

-for probationary 
teacher 70 58.8 78.8 

Number of 
observations per 

evaluation 22 18.5 24*8 

Beporting and 
Follov*up 

Post-conference 54 45.4 60.7 

Form of report 
statement under 

several headings 22 18.5 24*7 

unstructured 

statement 39 32.8 43.8 

ratings for various 

components 8 6.7 9*0 

flummary score 10 8.4 11.2 

Destination of 
report 

teacher 49 41.2 55.1 

school 49 41.2 55.1 

beard 52 43.7 58.4 

Follow-up 

plan developed 38 32.0 42.7 

monitoring of plan 29 24.3 32.6 
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Table 19 (continued) 



Component Number of Percentage Percentage 

Boards of All 119 of 89 

Responding Boards with 
Boards Teacher 
Policies 



Differentiation of procedures 

Differentiation 
between procedures 
(othe'- than frequency 
of eva uation) 
for peraanent and 
probationary 

teachers 31 26.0 34.8 

Differentiation 
between procedures 
for satisfactory 
and unsatisfactory 

teachers 42 35*3 47.1 

Explicit 

differentiation of 
appraisal system 

into two types 19 16.0 21.3 

Evolution of policy 
Development of 

policy 32 26.9 35-9 

Implementation of 

policy 27 2?. 6 30.4 

Review of policy 

and/or procedures 1*^ 14-3 19«1 
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Preparation 



Purposes 



Referring to the section on Purposes in table ly. we can see that, on the whole, Ontario 
boards are more concerned with teacher appraisal as a means for developing teaching staff and 
improving the quality of instruction then for providing information for various administrative 
decisions. Fairly typical of many boards' statements of purpose are the following (all 
quotations are from oi ninal policies, guidelines, or instruments; in order to protect the 
anonymity of the boards, no references are given): 

1. To encourage and maintain a high standard of performance for all personnel. 

2. To foster job satisfaction. 

3. To support the positive self-image and confidence of all personnel. 

4. To facilitate a high quality of educational services for students. 

5. To foster increased co-operative action on the part of all personnel. 

6. To provide a challenging work environment. 

and 

Evaluation of teacher pev'formance shall be approached as a positive activity 
designed U reinforce the daily achievements of teachers as they work with 
children aid young people. The process should generate insights about their 
instructional performance and their professional contribution that will enhance 
the development of teaching skills. 

A closer examination of the teacher policies confirmed the greater importance boards 
attach to non-administrative purposes for teacher appraisal. Among those boards whose 
materials included some statement of purpose, only one specified only administrative purposes. 
In all other cases where an administrative use for appraisal was stated, there was also 
explicit attention given to developmental /improvement uses. A fairly detailed statement of 
purposes, which illustrates dual concerns, follows: 



To provide, through formative and summative evaluation of teaching staff, 
every assistance toward the progress of our students by securing for them 
the most favouraDle conditions for growth and the achievement of 
excellence in their studies. 

To provide assistance for teachers in establishing the mosL desirable 
conditions for instruction. 

To assess instruction in the classroom and teacher contribution in the 
school so that strengths and weaknesses may be recognized and sympathetic 
direction given to teachers for the improvement of their professional 
practices. 




4. To identify teachers who have leadership potential and to assess the 
performance of teachers who aspire to leadership positions so that the 
selection process may be fair and equitable and the school system have the 
best leadership talent available. 

5. To provide the necessary formative and summative evaluation processes and 
statements for the improvement and/or dismissal of teachers whose 
performance falls below acceptable standards for the school system. 

6. To assess and improve the performance of teachers holding positions of 
responsibility and, ir necessary, provide processes and statements 
necessary for the removal of persons from such positions, if performance 
falls below acceptable leadership standards for the system. 

7. To provide supervisory officers with the necessary information to report 
to the Board on the performance of teaching staff and quality of 
instruction in the system, as required by Section 250 of the bducction 
Act 1974. 

8. To provide principals with the necessary information to fulfill their 
duties under Regulation 704/78; i.e., "supervise the instruction in the 
school and advise and assist any teacher, in co-operation with the teacher 
in charge of the organizational unit or program in which the teacher 
teaches; report to his board in writing, on its request, on the 
effectiveness of members of the teaching s;,aff. 



recommend to his board 



(a) the appointment and promotion of teachers and 

(b) the demotion or dismissal of a teacher whose work o>^ attitude is 
unsatisfactory, but only after warning the teacher in writing, giving 
him assistance and allowing him a reasonable time to improve." 

9. To provide a record of performance that will be helpful in responding to 
requests for references from other prospective employers. 

10. To provide a record of performance that will be helpful in making 
recommendations for various types of certification and entry into courses. 

11. To provide for teachers, at periodic intervals through the evaluation 
interview and summative evaluation statement, a clear understanding of 
their perceived level of performance in the school system and, thereby, 
assist teachers in setting performance objectives for improvement, if 
necessary; to enhance their feelings of self-worth. 



Criteria and Standards 



The largest single component of many boards' teacher policies is a listing of the criteria t 
be used in making judgements about teaching. (A very small number of boards indicate that 
school teaching staff should be involved in the selection of criteria in their schools, but, 
even among these boards, the policy may suggest possible criteria from which individual 
schools may choose. ) 
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The form in which criteria are stated varies tremendously among boards, ranging from very 
brief statements (e.g., "intorpersonal relationships", "planning and preparation") to lengthy 
lists of behaviours expected of a good teacher. Sometimes criteria are broad areas to be 
assessed by means of performance indicators and descriptors. At other times criteria are very 
specific indicators of competence in certain areas. As an example of the first usage, in one 
boa-^d's policy, when the criterion is "human relations", one of the "descriptors" is "control 
of students is maintained in a harmonious atmosphere". As an example of the second usage, in 
another board, in the area of teacher-student relationships, the expectation is expressed that 
"the teacher's personal demeanor creates mutual respect with students and encourages students 
to view the teacher as one genuinely interested in their welfare", and one of the criteria is 
that the teacher "is consistent and fair in expectations and reactions to student behaviour". 
Among various boards, there is mention of traits and qualities, of skills, competencies, and 
expectations, and there is some discussion of "presage", "process", and "product" types of 
criteria. This complexity is increased wher the notion of standards is introduced, with 
standards being for some exactly what criteria are for others. 

Accordingly, in analysing policies for this project, we decided to use the notion of 
"criterion domain". This notion would be able to incorporate criteria of varying specificity 
and type from responding beards. With criterion domain, the focus would be on the area or 
domain of teacher performance rather than on the specific criteria employed. 

At the same time, we decided to retain the commonly used term "standard" but to restrict 
its use to situations involving (1) the specification of how acceptable performance is to be 
measured, and (2) the level of attainment to be achieved. 

Referring to criterion domains in table 19 we can see that the six domains are not 
mutually exclusive. Classroom performance is the area most often mentioned in boards' 
statements of criteria for teachers, with interpersonal relationships (many of which take 
place within the classroom), school and community involvement, and professional development 
also listed by more than half of the boards. 

The use of personal qualities as criteria is somewhat less frequent (and very few boards 
rely exclusively on listings of Su^h qualities), and the teicher's contribution to pupils' 
religious development is restricted to some of the separate school boards. 
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None of the responding boards had teacher policies indicating explicit standards, when a 
standard was considered to relate to a measure rather than just an indicator. 
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Activities Prior to Appraisal 

Policies from 41 boards required that pre-evaluation conferences be held with their teachers. 
An additional four boards specified another method of notifying teachers about impending 
evaluations, such as by memorandum. 

Objective-Setting 

The expectation that teachers would set objectives was mentioned in the teacher evaluation 
documents of 38 boards. However, the nature and scope of objective-setting in the evaluation 
process varied considerably among boards. In some, objective-setting referred to activities 
surrounding a particular lesson to be observed by the evaluator; in others, objective-setting 
was an ongoing, cyclical process. An example of the former comes from a letter describing one 
board's system: 



We have undertaken to use the model, that was promoted by O.S.S.T.F., which is 
referred to as a clinical model, whereby teachers are requested to identify 
their lesson objecti ve(s) , their strategies, and the indicators that should be 
explained to the evaluator and be obvious in the classroom. 

The principal, vice-principal or department head (in the secondary school) 
visits the classroom and evaluates the lesson according to the objectives 
agreed upon with the teacher. Subsequently a post-visit meeting takes place in 
which the teacher and the evaluator talk about the lesson. 

In contrast, the material presented in figure 3 is adapted from the policy of a board in 
which objective-setting is a much longer-term matter. 



Figure 3: Objective-Setting by Teachers (Policy Excerpt) 

Approximate Action Required 

Month 

June Principal has a meeting with the teachers 

being evaluated to outline the procedures 
for evaluation. A discussion of Board and 
school objectives and expectations will take 
place at this time. 

September Teacher assesses the impact of the Board's 

and school's expectations as they pertain to 
their specific class's needs and abilities. 

Principal visits the classroom to familiarize 
himself with the teacher's needs and 
expectations. 

30 . . 
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Figure 3 (continued) 



Principal and teacher come to an agreement as 
to what expectation areas are to be evaluated 
that year. . . . 

0^*°^®^ Teacher drafts specific goals and statem ^.ts 

for tne chosen areas of evaluative.). 

Principal and teacher meet to draw up the 
formal docunent. 

October to May Principal organizes a series of forxal and 

informal visitations to the classroom of the 
teacher* • • • 



A final year end summary would be made based 
upon the results of the pr^^ncipal's 
visitations and observations of the new 
teacher that year. ... 

June and the Based upon the results c" the performance 

succeeding achievement summary, the principal will 

year(s) recommend appropriate inservice training or 

professional development to meet the needs of 
the teacher beyond that already given 
throughout the year.... 

SPECIAL NOTES 



1 • It is important to remember that this is an ongoing 

program. The Report in June should not be "filed away" 

but ought to be referred to regularly by both parties 

in succeeding years in order to assist in setting personal 

annual objectives and professional development. 



A similar approach from another board makes more explicit the cyclical quality of 
objective- setting: 



The basic cycle for the Evaluation for Professional Growth process involves a 
pre-visit conference, observation and a post-visit, conference... 

In addition to providing an opportunity for dialogue, the preparation of 
developmental and/or iirprovement goals and an action plan, the post-conference 
may also be used as a pre-conference for the next observation if this is to 
take place in the near future. The post-conference is a critical stage in the 
Evaluation for Profes sional Growth process. It must be meaningful to both 
parties, and a goal-oriented experience for the teacher; it will provide the 
basis for evaluator/teacher co-operation towards developmental and/or 
improvement goals. 

In some boards, too, objective-setting is listed merely as one of the criteria (e.g., 
"has well defined objectives and works toward them") to be met by a successful teacher; in 
others, the objective-setting process is the essence of the entire teacher evaluation process. 

The following example of the latter type comes from a board in which objective-setting 
happens to be exactly the same as that for administrators at all levels. The policy cites 
nothing about criteria or evaluation methodology, focussing simply on the process to be used: 
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The intent of this process i. '■hat each person in the system will, in 
co-operation with his/her imii Jiate supervisor, participate in the development 
of job oriented, self-oriented goals. 

This process provides the opportunity for each person to assess where he/she is 
in relation to the specific job expectations. Having done this, the individudl 
will establish a set of realistic goals for a given time period. These goals, 
once agreed upon, will form the basis for co-operative action on the part of 
the employee and supervisor. 

Throughout the process, those involved will monitor and, where necessary, 
modify the goals. At the end of the process, the stated goals will form the 
basis of a summative evaluation and recommendations for future action. 

In summary, objective-setting rmong the 38 boards varied considerably, and only 23 boards 
described their variant in some detail. All 28 boards that specified who was responsible for 
setting objectives indicated a collaboration between evaluator and evaluatee. 

Data Collection 

Methods 

As is clear from table 19, the primary mode of collecting data for teacher evaluation was 
classroom observation. Small numbers specified that interviews were used to collect 
information on teachers' activities about which the evaluator might not know and/or that 
documents (primarily teachers' lesson plans and students' classroom work) were used. 

Sources of Information 

In the teacher policies collected, there is very little suggestion that data about teachers' 
performance could be collected from others such as peers, parents, or students who would have 
information on this topic. Evaluators, it is evident, are thought to be able to make 
evaluative judgements through observing and/or talking to the teacher, although formal 
interviews between the evaluator and evaluatee are rarely specified. 

Evaluator(s) 

As is seen in table 19, principals are the group most often responsible for teacher 
evaluation, and the identity of the evaluator is one of the most often specified components of 
teacher appraisal systems. Although principals are most heavily involved, individuals in a 
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number of other roles may serve as evaluators under some circumstances, as -s outlined in 
table 20. In this table, a distinction is made between those who have primary responsibility 
for evaluating teachers and those who assist tfje primary evaluator or contribute as evaluators 
under particular conditions. 



l^^^^ ^Q- Number and Percentage of Boards Specifying Particular Primary and Contributi ng 

Evaluators in Teacher App raisal Policies 



Evaluator(s) 



Number 
of 

Boards 



Percentage 
of all 119 
Responding 
Boards 



Percentage 
of 89 

Boards with 
Teacher 
Pol icies 



Primary 








principal 


79 


66.4 


88.8 


vice-pri ncipal 


1 


0.8 


1.1 


superintendent 


18 


15.1 


20.2 


di rector 


1 


0.8 


1.1 


department head 


1 


0.8 


1.1 


other 


2 


1.7 


2.2 


Contributing 








evaluatee 


21 


17.6 


23.6 


vice-principal 


35 


29.4 


39.3 


superintendent 


27 


22.7 


30.3 


director 


7 


5.9 


7.9 


department head 


16 


13.4 


18.0 


coordinator 


1 


0.8 


1.1 


other 


2 


1.7 


2.2 



From table 20, we see that the only involvement of principals is as primary evaluator; 
they are specified as such by 79 boards. Superintendents also serve as primary evaluator in 
18 boards, but, from a closer examination of policies, appear never to be the sole evaluator. 
Some boards appear to use either the superintendent or principal as primary evaluator without 
specifying reasons for the involvement of one or the other group in particular situations. In 
other boards, however, superintendents may be involved in a different manner (e.g., less 
frequently) or under different circumstances (e.g., for classroom observation only) than 
principals are. A final situation in which superintendents are involved as primary evaluator 
is suggested by reference to table 19, where the category "differentiation between procedures 
for permanent and probationary teachers" is mainly comprised of cases in which the evaluator 
differs for these two groups of teachers. This category includes 31 boards. 
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The nature of the involvement of contributing evaluators, as listed in table 20, varies 
according to the group involved. Self-evaluation by the teacher typically occurs in advance 
of a formal evaluc.ion by another. Vice-principals or department heads, in general, are 
involved in providing various types of assistance to the primary evaluator, usually 
principals. Superintendents or directors, however, when considered in their contributing 
role, are typically asked to evaluate teachers when there is a difficulty of some type or if 
second opinion is desired for a particular reason. 



Time Spent 



Three-quarters of the boards included some sort of specification of the frequency with which 
formal teacher evaluation was to take place, but only one-quarter of them included a 
prescribed number of observations per evaluation. Table 21 includes, for permanent and 
probationary teachers separately, more detailed information about the precise frequencies 
prescribed. It is clear that evaluation is required much more frequently for the probationary 
group. Table 22 indicates the number of observations per evaluation that were specified by 
the 22 boards including this component in their policies. Although this component is rarely 
specified, when it is, the most common response is that three to four observations should be 
made for each evaluation. 



Table 21: Number and Percentage of Boards Specifying Particular Frequencies of Evaluation 
for Permanent and Probationary Teachers 



Frequency 



Permanent Teacher 

2 or more per 
year 

1 per year 
every 2 years 
every 3-5 years 
more than 5 years 

probationary Teacher 

2 or more per 
year 

1 per year 



Number of 
Boards 



3 

16 
14 
33 
1 



53 
17 



Percentage 
or all 119 
Responding 
Boards 



2.5 
13.4 
11.8 
27.7 

0.8 



44.5 
14.3 



Percentage 
of 89 

Boards with 
Teacher 
Pol icies 



3.4 

18.0 
15.7 
37.1 
1.1 



59.6 
19.1 
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^^^^^ — Number and Percentage of Boards Specifying Particular Numbers of Observations 
Per Evaluation for Teachers 



Nuaber of 
Observations 
Per Evaluation 



Number of 
Boards 



Percentage of 
All 119 
Responding 
Boards 



Percentage of 
89 Boards 
with Teacher 
Policies 



One 
Two 

Three to four 
More than four 



3 

12 
0 



5.9 
2-5 
10,1 

0.0 



7.9 
5.4 
13.5 
0-0 



Reporting and FollowUp 
Post-evaluation Conference 

Table 19 shows that 60.7 per cent of the boards providing teacher policy information specified 
a post-cvaluation conference between teacher and evaluator. 

Form of Report 

Sixty-four boards specified the precise format of the reports summarizing teacher evaluations. 
Table 19 indicates the numbers reporting the four variants identified. Some boards ware coded 
as belonging to more than one category. 



The most frequent sort of report, "unstructured statement", simply involved the 
evaluator's describing or commenting on those aspects; of teacher performance considered 
noteworthy. The next most frequent, "statement under several headings", was similar but 
required the evaluator to make comments on particular topics, which often corresponded to that 
board's criterion areas. An example of headings from one board are: Planning and 
Preparation, Instructional Practices, Evaluation Techniques, Classroom Management, Subject 
Competency and Professional Growth, Interpersonal Relationships, Contribution to Total School 
Effort, and Personal Characteristics. The report classified as "summary score" involved the 
evaluator's making a judgement of overall teacher functioning (e.g., excellent, very good, 
good, fair, weak, or unsatisfactory) and the report classified as "ratings for various 
components" involved assigning either qualitative grades or numerical scores to components of 
teacher behaviour. 
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Destination of Report 



Slightly over half the boards providing policy information specified the destination of copies 
of the reports summarizing teacher evaluations. In the majority of these boards, copies were 
to go to teachers who were evaluated, to their school or principal, and to the board offices; 
however, in some boards, copies were to go to only two of these three. In general, there was 
little detail regarding destination, with a very small number of boards mandating specific 
controls such as the destruction of reports after a cert£.in number of years or the restriction 
of use for purposes other than the immediate appraisal without the teacher's permission. 

Follow- Up 

Policies of 38 boards stated that a plan should be developed as a consequence of the teacher 
evaluation process; among these boards, only s^ en provided details about such a plan. 

One of the latter is a board which lays unusual stress on the remedial, training, and 
developmental aspects of the evaluation process. Its overall evaluation model is reproduced 
below: 

1. Self-evaluation 

Personal and individual process of examining 
one's performance within personal goals and 
those of the school and system. 

2. Collegial evaluation 

External assessment of individual performance 
by immediate superordinate with stress on 
positive aspects of professional growth. 
Evaluation to be based upon mutUdlly agreed 
upon criteria. 

3. Professional remediation 

Specific, immediate assistance to remedy 
identified individual problems or concerns. 

4. Inservice training 

Upon identification of common needs, short 
specific programs established at family of 
schools/board level. 

5. Professional development 

Ongoing, long range programs dealing with 
board at the board wide level. 



Establ ished 
and 

supported 
by individual 
schools 



Establ ished 
and 

supported by 
board and/or 
superintendents 
and/or 
principals 



36 S",) 



The policy manual for this board also includes a section containing ideas for meeting 
expectations and for in-service and professional development, as well as some samples of forms 
for teacher self-evaluation. Resources of this type are included in only a small number of 
boards. 

Just as it is unusual for boards to provide details regarding planning after evaluation, 
so it is unusual for them to specify that a post-evaluation plan for development or 
improvement should be monitored by a supervisor. Only 29 boards mentioned such monitoring; of 
these, only eight provided any specific information about how this would be done. 

The two following examples of arrangements for post-evaluation planning are taken from 
policies at the end of the continuum with the highest specification of the monitoring process. 
Their brevity suggests how little attention, overall, is given to the monitoring of 
post-evaluation plans. From one board: 

During the visitation period, the evaluator will discuss his/her observations 
with the teacher. Should there be areas where improvement is required, the 
evaluator shall make specific recommendations. Any or all of the following 
personnel may be involved in the development and implementation of a program to 
assist the teacher: other teachers, Co-ordinators/Consultants, Department 
Heads, Vice-Principals, Principals, Superintendents and other resource persons. 

From the other: 

The process does not end with the post-visit conference. The evaluator and 
teacher will continue to work together in an ef f o) t to meet the improvement 
and/or developmental goals and to implement the action plan. Throughout the 
process, the evaluator must consider such things as the needs and previous 
experience of the teacher, the expectation of the school/system, and the role 
of the evaluator in providing assistance to the teacher. 

Differentiation of Procedures 
Permanent and Probationary Teachers 

This category, which in table 19 contains 31 boards, was described earlier in this section 
under Evaluator(s). Most differentiation of procedures involves different evaluators used for 
permanent and probationary teachers and different frequencies of evaluation. 

'5^ 



Satisfactory and Unsatisfactory Teachers 



The policies of 42 boards differentiated, to some degree, procedures to ba followed with 
teachers who were found to be performing at a satis. actory level and with those who were not. 
The procedures ranged from simple ones described in only a brief paragraph to complex policies 
a half-dozen pages in length. 



Fourteen of these boards made rather brief reference to the fact that there were special 
arrangements to be made for teachers who were experiencing m-jor difficulties and did not 
label or describe in detail such arrangements. For instance, in one board, when a teacher is 
identified as "unsatisfactory". 



A written plan for improvement is to be created by the teacher and the 
evaluator. The actions taken to effect improvement will be monitored by the 
immediate supervisors within a definite time period indicated. Further 
evaluation by the immediate supervisors and an evaluation by a 
Superintendent is required within a specified time. 



^.*om another board, the supervisor/evaluator is advised to consider alternatives: 



If required, discuss the following alternatives: 

(1) Promotion 

(2) Positions of added responsibility 

(3) Reassignment 

(4) Transfer within/outside the system 

(5) Retraining 

(6) Demotion 

(7) Early retirement incentives 

(8) Termination 



Twenty-four of the 42 boards had more detailed descriptions of procedures to be followed 
in the process variously termed "on review", "under review", or "documentation". Among these 
boards, also, there was substantial range in complexity. Typical of the process required in 
many boards is the following: 



ON REVIEW PROCE^ ;RES 



Procedures 

1. The teacher will be informed in writing that his performance is not 

meeting acceptable standards as defined by the Ac't and in the expectations 
of teachers as stated in Board Policy. 
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2. The teacher will be informed in writing as to what is required to meet 
acceptable standards. 

3. The Superintendent will be consulted and asked to appoint a team to 
monitor the teacher's performance. 

4. Teachers in chare's of organizational units, Consultants and Vice- 
Pnncipal(s) may be asked to assist the teacher to meet acceptable 
standards. 

5. The teacher will be given a reasonable period to improve. 

6. The evaluation reports and the recommendation of the Principal and/or 
Vice-Principal(s) will be considered by the Superintendent in making a 
recommendation to the Board. 

7. The teacher will be informed in writing of the decision of t.ie 
Superintendent, i.e.: 

(a) return to normal supervision and evaluation routine; 

(b) continue "on review"; 

(c) recommend termination by Board motion. 

Some of these policies included a description of an appeal procedure, while others did not. 



Four of the 42 boards had extremely complex and detailed procedures for dealing with 
teachers with difficulties. A sample of one prccedure is giver in figure 4. 

^^'g^^^ ^- Evaluation Procedures for Teachers Whose Performan ce Is Unsatisfactory 

(Policy Excerpt) ^ 



Special Evaluation Procedures for Permanent Teachers Having Difficulty 
Letter of Concern 

1. A teacher on permanent contract who is having serious difficulty and has had at 
least two visits for evaluation shall receive from his principal or Area 
Superintendent a signed Letter of Concern. 

2. The Letter shall point out the teacher's problems, make definite recommendations for 
improvement, and assure the teacher* that help is available. When the Letter is 
presented to the teacher, the principal or Area Superintendent shall discuss it with 
him fully and, if he thinks it advisable, invite another supervisor to attend the 
discussion. 

3. The elementary school principal or Area Superintendent concerned shall send a copy 
of the Letter to the Superintendent of Operations. 

4. The secondary school principal or Area Superintendent concerned shall send a copy of 
the Letter to the Superintendent of Operations, wait three teaching days after 
issuing it to give the teacher in question an opportunity to request, in writing, 
that the O.S.S.T.F. not be informed, and then, if the teacher has not so requested 
notify the Executive of O.S.S.T.F., District 4, that he has issued a Letter of 
Concern to him. 

Letter of Doubt 

1. A teacher given a Letter of Concern shall have his work evaluated further. If, 
after at least thirty teaching days from the date when he received a Letter of 
Concern, the teacher has had at least two evaluations of his work but continues to 
show little or no improvement in spite of determ- ned efforts to assist him, he shall 
receive from his principal or Area Superintendent a signed Letter of Doubt 
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Figure 4 (continued) 



2. The Letter of Doubt shall point out in specific terms the teacher's continuing areas 
of difficulty, suggest possible solutions for them, and state that unless definite 
improvement occurs, the cont/act may by terminated. 

3. The elementary school principal or Area Superintendent concerned shall discuss the 
Letter with the teacher in the presence of another supervisor, keep a copy for his 
records, and send a copy to the Superintendent of Operations. 

4. "he secondary school principal or Area Superintendent concerned shall discuss the 
Letter with the teacher in the presence of another supervisor, keep a copy for his 
records, send a copy to the Superintendent of Operations, wait three teaching days 
after issuing the Letter to give the teacher an opportunity to request, in writing, 
that the O.S.S.T.F not be informed, and then, if he has not so requested notify the 
Executive of O.S.S.T.F., District 4, that he has given him a Letter of Doubt. 

Letter of Recognition of Immediate Improvement 

If the principal or Area Superintendent believes that a teacher given a Letter of 
Doubt has, in fact, overcome his stated problem, he shall supply him with a Letter 
of Recognition of Immediate Improvement and send a copy to the Superintendent of 
Operations, and this copy will be placed in the teacher's evaluation file. The 
teacher in .question shall then proceed on the salary grid the following 1 September. 

Letter of Further Consideration and Letter Recommending Termination of Contract 

A teacher given a Letter of Doubt shall have his work evaluated further. If, after 
at least thirty teaching days from the date when he received a Letter of Doubt, 
marked improvement does not occur, the principal or Area Superintendent shall give 
him a Letter of Further Considerat jn or a Letter Recommending Termination of 
Contract, depending on the degree to which his service has been less than 
satisfactory. 

Letter of Further Consideration 

A Letter of Further Consideration shall outline the teacher's difficulties, make 
clear suggestions for improvement, and describe very definitely any changes 
recommended. 

1. The principal or Area Superintendent shall prepare the Letter in triplicate, sign, 
and have the teacher sign, all three copies, keep a copy, give the teacher a copy, 
and send a copy to the Superintendent of Operations. He shall also discuss the 
contents of the Letter with the teacher in detail in the presence of another 
supervisor. 

2. Transfer of the teacher to another school or other action concerning placement may 
be considered. 

3. A teacher given a Letter of Further Consideration will not advance on the salary 
grid, effective 1 September of the next school year. When his progression on the 
grid' is to be withheld, the teacher must be notified before 1 April of the Board's 
intention to withhold the progression and shall receive a written statement of the 
reasons why his service is not considered satisfactory. However, his salary 
progression shall not be withheld for more than two consecutive years. 

Letter Recoounending Termination oi Contract 

1. Before a teacher receives a Letter Recommending Termination of Contract, the 
following requirements must have been satisfied: 

(a) the teacher has had at least six documented supervisory visits; 

(b) at least one supervisor other than the principal has discussed the teacher's 
problems with him; 

(c) the teacher has received both a Letter of Concern and a Letter of Doubt; 
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(d) in the case of a secondary school teacher, the principal has required the 
Department Head to assist the teacher in every way possible and has on file a 
report from the Head indicating the assistance he has given the teacher; 

(e) the principal and Area Superintendent have discussed the case with the 
Superintendent of Operations. 

2. A Letter Recommending Termination of Contract must state clearly the reasons for its 
issuance, review all areas of unsatisfactory performance, summarize the specific 
efforts made to assist the teacher, and notify him specifically that a 
recommendation will be made to the Board to terminate his contract. 

3. The principal or Area Superintendent shall prepare the Letter in triplicate, sign, 
and have the teacher sign all three copies, keep a copy, give the teacher a copy, 
and send a copy to the Superintendent of Operations. 

4. Principals or Area Superintendents 

(a) must notify the teacher, in writing of their intention to recommend his 
dismissal to the Board, which may confirm the recommendation by 30 November, to 
take effect on 31 December immediately following or by 31 May, to take effect 
on 31 August immediately following; 

(b) must, in case of a secondary sc' .ol teacher, notify the Board of Regents of 
O.S.S.T.F., of the circumstances under which the teacher left their staff. 

Exceptional Situations - Permanent Teachers 

Although the special evaluation procedures outlined in this policy statement for dealing 
with permanent teachers having instructional difficulties form the regular evaluation 
sequence in such cases, exceptional situations may require exceptional handling. A very 
serious instructional or other problem occurring during the school year may require 
giving a permanent teacher a Letter of Concern followed by a Letter of Doubt not sooner 
than fifteen teaching days after the issuance of the Letter of Concern, or g Letter 
Recommending Termination of Contract not sooner than fifteen teaching days after the 
issuance of a Letter of Doubt. 

The principal and Area Superintendent concerned should discuss all such situations 
with the Superintendent of Operations and advise the appropriate affiliate. 

Special Evaluation Procedures for Probationary Teachers Having Difficulty 

Teachers on probationary contracts and experiencing difficulty will, wherever 
possible, be evaluated in the same sequence outlined above. 

Notwithstanding the Board reserves the right to terminate the contract of a 
probationary teacher according to the terms of the contract and the provisions of the 
Education Act. 



Explicit Differentiation of Appraisal System into Two Types 

Table 19 indicates that systems in 19 of the boards involved explicit differentiation of 
teacher evaluation into two types which depended on different purposes and/or facets of 
teacher evaluation. These distinctions, which on the whole tended to be made in the more 
highly elaborated policies, are interesting in and of themselves and because they illustrate 
problems with terminology and Definitions, which make comparisons across boards very 
difficult. 
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Four sets of distinctions are discussed: formative vs. summative; administrative vs. 
non-admini strati ve; supervision vs. evaluation; and classroom vs. comprehensive. 

The most commonly made distinction was between formative and summative processes and the 
procedures associated with these. However, these terms mean sl-ightly different things to 
different boards. 

Several agreed that formative refers to an ongoing developmental process, while summative 
occurs after a defined period has elapsed. For example, from one board: 

Formative Evaluation is an ongoing developmental process directed toward 
improved performance and job satisfaction. 

Summative Evaluation provides a clear perception of the overall effectiveness 
of an individual's performance. It "sums up" comments on all aspects of 
performance observed over a definite period of time. 



And from a similar one, 



Summative evaluation of a curriculum, program, student, teacher or 
administrator judges the effectiveness of the learning, instruction, or action 
in relation to some standard of excellence after the learning, instruction or 
action has taken place 

Formative evaluation occurs during the process of curriculum and program 
construction, learning, teaching and administration for the purpose of guiding 
the development or altering these processes during the formation stage. 

However, other boards attach other meanings to the terms. For examp'i , in one, formative 
tends to mean informal and summative to mean formal: 



Formative : Formative assessment is the on-going, informal approach to the 
improvement of instruction. It should be viewed as a developmental function; a 
counselling tool that can help "coach" a teacher toward better performance. It 
requires openness and two-way communication for the parties involved. 
Formative assessment may take one of two forms: 

Sel f Assessment : a process wherein an individual teacher might assess 
hiT/her own performance against the prescribed criteria outlined in the 
Board's Role Description for Teachers, or against his/her personal 
objectives for the year. 

Collegial Assessment : a process wherein a colleague (a fellow teacher. 
Department Head, Principal or Vice-Principal) might assist the teacher in 
analyzing and providing feed-back about the teacher's performance. This 
type of assessment might be initiated by either the teacher or an 
administrator. In either case, the assessment is of a formative nature 
and no permanent records are kept. The primary focus of such a process is 
to assist the teacher in the improvement of instruction. 

Su mmative : Summative assessment is the more formal approach to an annual 

performance review. , 
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Another board uses formative and summative in ways that correspond with another 
distinction (between an administrative evaluation process and a non-administrative one) to be 
described more fully below. For example, 



Formative reports identify strengths in the teacher's performance and areas for 
improvement or further development, with specific time-lines included. 

Summative reports are t'omprehensi ve and summarize observations of the teacher's 
total performance for purposes of promotion, selection, exchange, recognition 
or as required. 

This example also suggests a further distinction, to be elaborated later in this section, 
between comprehensive evaluation and classroom evaluation. 



In several boards there is a distinction between an administrative process and a 
non-administrative process (variously called "improvement", "instructional", or "professional 
growth"), and the two processes entail separate procedures. For example, 

EVALUATION FOR PROFESSIONAL GROWTH . .■. 

-will be directly related to the professional development and/or 
improvement of teachers 

-will involve each teacher in the process within a three year cycle 

ADMINISTRATIVE EVALUATION . . . . 

-will involve all probationary teachers at least twice in each year of the 
probationary period of the contracts 

-tends to be summative in nature and requires indication of the evaluator's 
(usually the principal's) support or non support for the teacher in his or 
her role or potential role in the school and/or system 

-is essentially the responsibility of the principal and vice-principal (The 
superintendents may become involved in the decision-making proces'..) 

-is clearly separated from the process of Evaluation fo r Professional 
Growth 

-will NOT involve department heads, assistant heads or consultants/ 
co-ordinators as evaluators in leading to administrative decisions. This 
group of personnel may assist evaluators and teachers but will not provide 
verbal or written reports on teacher performance when involved in this 
process. 

Closely related to both of the above terminological distinctions is that made between 
supervision and evaluation. 
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Several boards agree that supervision refers to a process involving giving helpful 
support to a teacher, while evaluation involves making a judgement. The first example below 
shows the close relationships among evaluation, summative, and administrative: 



Supervision 

Supervision is a process whose primary function is the improvement of a 
limited number of aspects of the teacher's work.... 

The result will be a report designed to be helpful with suggestions as to 
how the teacher's work can further improve in the future 

Evaluation 

Evaluation is the process of making a judgmant about the overall quality 
of a teacher's work. Its primary function is to assist in making 
administrative decisions about the teacher* s future: promotion, demotion, 
retention, or dismissal. 

The following example again shows the closeness of the terms evaluation and 
administrative, and again hints at the distinction between classroom and comprehensi 
processes: 



Decisions related to the evaluation of teachers are of two types: 

SUPERVISORY decisions, regarding the responsibilities of teachers within the 
classroom. 

SUPERVISION refers to the activities of the administrator and/or teacher which 
result in SUPERVISORY decisions. 

ADMINISTRATIVE decisions regarding the selection, placement, promotion, 
retention, demotion, and termination of staff. EVALUATION refers to the 
activities of the administrator which result in ADMINISTRATIVE decisions. 



There are also idiosyncratic uses of the terms supervision and evaluation. For example: 



TEACHER SUPERVISION : is that process or observation, review, discussion, and 
follow-up applicable to a teacher on permanent contract. Emphasis will be on 
improvement of the quality of teaching and learning, encouragement of personal 
professional growth, and the provision of information relative to changes in 
teaching assignment. 

TEACHER EVALUATION : is that process of observation, review, discussion, and 
follow-up applicable to a probationary teacher. Emphasis will be on criteria 
to be used in decisions regarding the granting of a permanent contract. 



Finally, a distinction between classroom and comprehensive evaluation is made expli 
a few boards. For e;^ample, from one board: 
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Formal teacher evaluation is divided into two parts: 

I. The evaluation of Classroom Performance . 

II- The Comprehensive Evaluation of the teacher's contribution to the 
total educational needs of the c* ild, the school and the Board of 
Education. 

The second example also overlaps the formative-summati ve distinction: 

Two types of evaluation are proposed: 

1. The Classroom Effectiveness Report (CER) formative evaluation 
composed of ongoing specific review of the teacher's classroom 
behaviour; 

2. The Professional Growth Review (PGR) — summative evaluation 
comprising a comprehensive review of all aspects of the teacher's 
activity as an educator. 

The four sets of distinctions (summative/formati ve , administrative/non-administrative, 
supervision/evaluation, and classroom/comprehensive) considered in this section involve terms 
in very common 3e in the province's stated policies and procedures for the performance 
appraisal of teachers. The fact that there is such diversity among boards in usage of the 
terms makes cross-board comparisons and the assignment of boards to a small number of 
categories or "types" with similar systems very difficult. 

Evolution of Policy 

Development 

Only 32 boards, as is seen in table 19, included in their documents information about how 
their teacher evaluation policy was developed. Most typically, they listed the composition of 
the committee involved. Less frequently, they also described one or more of the following: 
the historical situation leading to the committee's formation; the committee's mandate; and 
the committee's activities. There were occasional references, too, to the sources (e.g., 
other boards, outside experts, literature) consulted or used in policy development. Finally, 
a few boards described the experimental use draft policy materials in a small number of 
pilot schools. 




Implementation 



As table 19 shows, only 27 boards made any rrention of how newly developed teacher evaluation 
policies were to be implemented and, of these, few were any more specific than to say there 
would be "an inservice program to inform all principals and teachers". A few did specify who 
would be responsible for providing training for evaluators and evaluatees. In general, there 
was a certain amount of awareness tha . an implementation strategy was necessary, but very 
little detail about it. 



Review 



Only 17 of the boards provided any information about the mechanism for reviewing their teacher 
evaluation policy. The information ^ ds even less detailed than that provided with respect to 
the policy's development and implementation, with the exception of three boards. These 
boards, in contrast to the others who merely indicated that a method for reviewing their 
policy was necessary, had very specific plans for doing so. An example is given in figure 5. 



Figure 5: Review of Teacher Evaluation Policy (Policy Excerpt) 
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Plan for Review of the Model of Supervision 

Purpose of the Review Process : To determine whether the model of supervision as set up 

and implemented in our system is functioning as intended. 

Four stages will be used in determining the appropriateness of the model. 

STAGE 1: Presentation 

(a) Objective- To evaluate the level of understanding of the model gained from the 

presentation. Understanding is vital in the areas of: 

-phi losophy 
-process 

(b) Timeline: This stage of the review process will be conducted after the initial 

in-service session. 

(c) Tools: One or more of the following may be used: 

- quest i onna i re - i nterv i ews 

-written comments -co-operative evaluation 

(d) By whom: Supervisory Study Committee 

(e) Follow-up of Review: 

Further in-service as needed. 
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Figure 5 (continued) 

STAGE ?: Implementation 

(a) Objective: To evaluate the model to determine areas that need clarification, 

amplification or change. 

(b) Tiiieline: This stage of the review process will be conducted at the end of the 

first year of implementation. 

(c) Tools: Same possibility as STAGE 1. 

(d) By whom: Supervisory Study Committee 

(e) Follow-up of Review: 

Revisions to the model will be made as recommended by the committee. 

STAGE 3 

(a) Objective: To evaluate the model after it has been used for an extended period 

of time, to determine what extent the Supervisory model has 
contributed to the growth and development of the people involved and 
to the improvement of education in the Scnool System. 

(b) Timeline: This stage of the review process will be conducted 2 years after 

implementation. 

(c) Assessment Techniques: 

To be determined by the Supervisory Study Committee in advance of 
implementation. 

(d) By whom: An Ad Hoc committee, consisting of at least one representative from 

the Supervisory Study Committee. 

(e) Follow-up of Review: 

Recommendations to be brought to the Supervisory Study Committee as a 
whole. 



STAGE 4 



A formal review, once every five years, is to be conducted to evaluate the Policy, the 
procedures supporting the Policy and the perceived effects of the supervisory process as 
defined in the Policy. 



Cluster Analysis 

This technir,ue was used to determine whether or not teacher appraisal systems could be grouped 
into several distinct types, with each type possessing a particular set of system components. 
Neither the 30-component nor the 11-component cluster analysis technique yielded clear 
groupings. This failure may be partly attributable to the differences in boards' uses of 
various te>^ms, as described. However, the major reason, more probably, is that policies are 
aggregates of independent components, with different parts of the policies being based on 
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different assumptlonb. That is, there are nn sets of clearly defined, mutually exclusive 
frameworks that underlie the structures of teacher appraisal procedures as currently practised 
in Ontario. 



Principal Appraisal Policies 



Among the 119 boards that supplied information about staff appraisal, 63 provided some type of 
information about their policies for principals. In table 23, various components that may 
occur in principal policies are listed; for each component, the number and percentage of 
boards whose policies include that component are given. Percentages are expressed relative to 
all 119 responding boards and to the 63 boards which submitted principal information. The 
hierarchical organization of the table is very similar to that of table 19, which dealt with 
teachers. However, there are a few specific differences in categories, particularly under the 
heading Criterion Domains, between the two tables. 

Table 23: Number and Percentage of Boards Specifying Particular Components in Principal 
Appraisal Policies 



Compc ""t Number of Percentage of Percentage of 

Boards All 119 63 Boards 

Responding with Principal 

Boards Policies 



Preparation 

Purposes 

development/ 
ioprovement : 

general 4 3 '4 

development/ 
improvement : 

specific 30 25.2 

administrative : 

general 7 5«9 

administrative : 

specific 17 14*3 

routine 2 1.7 



6.3 

47.6 

11.1 

27.0 
3.2 
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Table 23 (continued) 



Component 



Nuffloer of Percentage of Percentage of 

Boards All II9 63 Boards 

Responding with Principal 

Boards Policies 



Criterion domains 
general 

administration 43 

school and 
communi ty 

relations 39 

program 
organization, 
development, and 
implementation 42 



personnel 
management , 
including 
evaluation and 
supervision 



contribution 
to board 

contribution 
to religious 
education 

Standards 
(explicit) 

Activities prior 
to appraisal 

notification 
in person 

notification 
by memorandum 

request for 
statecent about 
school 



42 
25 

13 



13 



request for 
self-evaluation 5 

pre-conference 36 

Objective-setting 47 



36.1 



32.7 



35.3 



35.3 
21.0 

10.9 
0.0 



4.2 
5.0 

10.9 

4.2 

30.3 
39.5 



68.3 



61 .9 



66.7 



66.7 
39.7 

20.6 
0.0 



7.9 
9.5 

20.6 

7.9 
57.1 
74.6 
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Table 23 (continued) 



Component 



Number of 
Boards 



Percentage of 
All 119 
Responding 
Boards 



Percentage of 
63 Boards 
with Principal 
Policies 



Data Collection 

Tata collection 
methods 

observation 

examination of 
docunents 

interview 

other 



16 

11 
39 
3 



Sources of infonnat: n 

principal 
being 

appraised 43 

teachers 12 

department 

heads 5 

vice-principal 4 

superintendent 3 

parents 5 

students 4 

others 2 

Appraiser(s) 

principal 
being 

appraised 22 

principal* s 
superintendent 46 

other 

superintendents 11 



director 

others 

evaluation 
team 



10 
6 

13 



15.1 

9^2 
32.8 
2.5 



36.1 
10.1 

4.2 
3.4 
2.5 
4.2 
3.4 
1.7 



18.5 

38.7 

9.2 
8.4 
5.0 

10.9 



28.6 

17.5 
^19 
4.8 



68.3 
19.0 

7.9 
6.3 
4.8 
7.9 
6.3 
3.2 



34.9 

73.0 

17.5 
15.9 
9.5 

20.6 
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Component 



Number of Percentage of Percentage of 
Boards All 1 19 63 Boards 

Responding with Principal 
Boards Policies 



Time spent 

frequency 27 

number of 

days 9 

Reporting and follov*up 

Post-conference 33 

Form of report 



statement under 

several 

headings 



unstructured 
statement 

ratings for 

various 

components 



11 
16 

5 
2 



summary score 
Destination of report 

principal 20 

board 25 
Follow*up 

plar developed 20 



monitoring 
of plan 



12 



Differentiation of procedures 

Differentiation 
among procedures 
for different 
categories of 
principals 7 

Differentiation 
between procedures 
for satisfactory 
and unsatisfactory 
principals 3 

Explicit 
differentiation 
of appraisal system 
into two types 5 



22.6 
7.5 

27.7 



9-2 
13.4 

4.2 
1 .7 

16.8 
21 .0 

16,8 
10.1 



5.9 



2.5 



4.2 



42.9 
14.4 

52.4 



17.5 
25.4 

7.9 
3.2 

31.7 
39-7 

31.8 
19.0 



11.1 



4.8 



7.9 
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Table 23 (continued) 



Component 



N' nber of 
Boards 



Percentage of 
All 119 
Responding 
Boards 



Percentage of 
63 Boards 
with Principal 
Policies 



Evolution of policy 



Developmenc of 
policy 



11 



9.3 



17.4 



Implementation 
of policy 



5.9 



11.1 



Review of policy 
and/or procedures 8 



6.7 



12.7 
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Preparation 



Purposes 



As is seen in table 26, Ontario boards are concerned with appraisal somewhat more as a method 
for improving principal performance than as a means for informing administrative decisions. A 
closer examination of principal policies showed that there were on' two cases where boards 
specified only administrative purposes, while there were a number of boards specifying 
improvement policies only. However, table 23 also indicates that the discrepancy between 
developmental/improvement purposes and administrative purposes is much less marked for 
principal policies than for teacher policies. Whereas 75.3 per cent of t .se boards v^ith 
teacher policies included statements about specific developmental /improvement purposes and 
36.0 per cent about administrative purposes, the analogous figures for principals are 47.5 per 
cent and 27.0 per cent. In addition, among the principal policies, there are more statements 
about general administrative purposes than about general developmental/improvement purposes, 
and this is not the case for teacher ,)olicies. 

In addition, a much smaller percentage of boards -ake any statements at all regarding the 
pu.-poses of principal appraisal. Among the boards that do, Hi.cussion is generally briefer 
than is the case for teachers. Typically succinct is the following statement which includes 
both improvement and administrative purposes: 

The Board recognizes its responsibility to provide an evaluation program which 
wi 1 1 

-encourage and support the practices of the successful principal, 

-identify areas of future development for each principal and assess the 
principal's growth in leadership, 

-develop an a'-curate record of each principal's accomplishments as a basis 
upon which future assignments can be made, 

-provide an accountability system to assure quality leadership. 

Another policy, which provides a little more concentration on the 
developmental/improvement type of purpose and a little less on the administrative type, has 
only a slightly longer discussion of purposes and yet is among the .nost elaborate in the 
province: 
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The fundamental purpose of an evaluation is to give the Principal/Vice- 
Principal an opportunity to examine his/iier philosophy, role and administrative 
practice in order to make any adjustment necessary to improve performance. The 
career needs should be satisfied both in the detailing of areas of strength and 
in identifying areas wh«re improvenfient could be made. A natural result of this 
exercise is the satisfaction that can be Jerived from the recognition of a job 
well-done by the Principal/Vice-Principal involved. The document provides a 
vehicle for ensuring that each Principal/Vice-^rincipal has a relatively 
current reporting of his/her performance as an administrator available both for 
personal and system use. 

The selection of the criteria permits the parties to the evaluation to exercise 
their right to ensure that a fair evaluation occurs. This also provides the 
latitude necessary to accommodate the wide variety both of the roles and of the 
personal ities involved. . . . 

In summary, it is intended that this document provide for recognition of fine 
performance and opportunities to improve where appropriate. As a result, 
students in (the) County will be better served. 



Criteria and Standards 

We see in table 23 that, among the 63 boards providing principal information, there is 
significant attention given to specifying criteria but there is no evidence of rigorously 
defining standards of expected performance. 

As with teacher policies, a very substantial pan of some principal policies is devoted 
to the general matter of criteria. As with teacher policies, also, there was some diversity 
regarding the usage of the term criteria, and so we used the notion of criterion domain once 
more. Regardless of exactly how the term criteria was used, there was considerable uniformity 
among boards in their general approach is this area. In most cases where criteria were 
considered, boards specified, under several domains, a large and diverse list of the 
behaviours expected of a responsible and successful principal. An example appears in 
figure 6. 

Figure 6: Criteria for Principal Appraisal (Policy Excerpt) 



cri:eria 



The criteria identified in this evaluation material seek to pinpoint the 
Principal' s/Vice-Principal's role in both the school and the system. The evaluation will be 
focussing attention on the role of the Principal/Vice-Principal as an educational leader and 
the sound judgment which he/she exercises in the day-to-day operation of the school. 

The fallowing list is .not designed in priority order, nor is it intended to be all-inclusive: 



ERIC 



Figure 6 (continued) 

1. Educational Leadership 

-has developed a clear statement of the educational philosophy of the school in its 
community, along with a written set of aims and objectives for that school 

-assists in developing, implementing and supporting Board policy/procedures 

-maintains affiliation with professional organizations 

-organizes effective procedures which result in the selection of quality personnel 
-provides for orientation and development of new staff 
-assigns appropriate responsibilities to staff members 

-encourages the participation of staff in the day-to-day management of the school 

-provides for tna effective use or Professional Activity Days 

-accepts responsibility for assisting in the training of student teachers 

2. Relations With Staff 

-has a clear and effectively-implemented program for the evaluation of staff 

-supports aid encourages staff members in their work 

-creates and maintains positive staff morale 

-is readily available for consultation by staff 

-assigns duties and classroom assignments equitably 

-develops and trains staff for positions of responsioi lity 

-provides opportunity for staff involvement 

-provides for in-service training programs and encourages intervisitation 
-provides direccion to staff regarding school procedures (i.e., school manual) 
-keeps staff well informed 

-has a positive and effective working relationship with internal and external support 
staff 

3. Relations With Students 

-has a clear and appropriate philosophy and strategy for the evaluation of students 

-endeavours to keep the students informed of the evaluation procedures 

-gives recognition to students who achieve excellence in curricular and extra-curricular 
programs and recognizes enthusiastic participation and improvement 

-is readily available to the student body 

-demonstrates a concern for students 

-provides support for students who require additional help 

-encourages and provides for optimum student participation in the total school program 
-has an effective and meaningful student activities program 

-has clearly-established and implemented guidelines for student deportment and discipline 
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Figure 6 (continued) 

-uses sound judgment when disciplining students 
-provides for orientation of all new students 

4. Relation With the Community 

-employs effective and well-defined procedures for reporting to parents 

-interprets effectively and clearly the school programs and policies (i.e., evaluation) to 
the public and provides opportunity fo.^ community feedback 

-fosters positive parent-teacher relations and encourages parent participation 
-effectively resolves parent-teacher conflicts and misunderstandings 
-co-operates with other public agencies related to the community and youth 
-encourages the utilization of community resources in the classroom 
-encourages and monitors community use of sc^^ools 

5. Instructional Program 

-has an awareness of and adheres to Ministry/Board guidelines 

-encourages and supports teachers ir the development and effective use of a variety of 
techniques and teaching strategies to meet student needs 

-works closely with the staff to implement and evaluate curricula 

-effectively develops a program to meet the needs of the students and provides a timetable 
to accommodate this program 

6. Integration 

-supports the "Family of Schools" approach 

-strives for close communication with associated schools to enhance the kindergarten to 13 
concept in curriculum and program 

7. Management of Plant 

-submits reports re site, facilities, alterations and renovations 
-provides for the care, storage and maintenance of all school equipment 
-attempts to maintain security and minimize acts of vandalism 

-monitors the effectiveness of the custodial service and the general maintenance of the 
school grounds 

-makes effective use of school facilities 
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Activities Prior to Appraisal 

Thirty-six boards specified that a conference be held prior to ^ne appraisal of a principal. 
Although a similar specification was made for teachers in 41 boards, the requirement of 
pre-conferencing was relatively more widespread (57.1 per cent) for principals than for 
teachers (46.1 per cent) when expressed as a percentage of boards submitting information for 
the role rather than as a percentage of all responding boards. 

There was somewhat more formal specification of the methods for beginning the principa'i 
appraisal process than was the case for teachers. Thirteen boards indicated that principals 
were asked for a statement about their school; five that they were notified in person; six 
that they were notified by memorandum; and five that they were asked for a self-evaluation. 

Two of these specifications are particularly important features in distinguishing some 
principal policies from teacher policies, and hence require further comment. 

The policies or procedures of 13 boards specified that particular kinds of information 
about the school be submitted by the principal to the appraisers. The type of information 
required ranged from fairly simple listings of school objectives or staffing to the complex 
packages required for submission to evaluation teams (see the section on Evaluation Teams 
below) or the detailed 22-page questionnaire specified by one board that contained eight 
sections: Philosophy, Evaluation, Supervision, School Organization, Administration, 
Relationship of School with Community, Future Plans, and Special Projects. 

Although a large number of boards suggested that some self-appraisal be carried out by 
principals, only five outlined in detail the precise technique to be used and its exact role 
in the appraisal process. 

Objective-Setting 

Objective-setting was an integral part of the performance appraisal process for principals in 
47 boards, i.e., 74.6 per cent of all boards submitting principal policies. In contrast only 
38 boards, 42.7 per cent of all those submitting teacher policies, had objective-setting as a 
component of their written teacher policies. 




In figure 7, a description of the objective-setting process arawn rrom one Doara s poncy 
for principals illustrates 'everal important features which are commonly specified. 



Figure 7: Objective-Setting by Principals (Policy Excerpt) 



TIME THE PRINCIPAL PERFORMANCE REVIEW MODEL 

FRAME 

SELECTION OF OBJECTIVES 

1. The principal considers and selects possible objectives for the school year. 

April Input is to be obtained from: 
May 

-the role statement 
•the self-evaluation 
-System/region objectives 
-the staff 
-the community 

2. The principal in consultation with the local s 'pervisory officer discusses the 
possible objectives and receives his input including: 

May -super intendency objectives, 

Sept. -a general review of principal's performance, the evaluation of objectives realized 
in the previous year. 

Sept. 3. The principal drafts a statement of objectives for the coming year on the 
Statement of Objectives Form according to the following directions: 

-a number of objectives as agreed upon by the supervisory officer and principal, 

-specific plan of action to be indicated, 

-data sources to be indicated in plan of action, 

-time line to be indicated, 

-a copy to be sent to the supervisory officer by September 30. 

COOPERATIVE AGREEMENT TO OBJECTIVE STATEMENTS 

April 1. The Statements of Objectives are the focus for discussion between the 
May principal and the supervisory officer. Tha statements vis-a-vis intent, priority, 

June approach, assessment data and time lines should be clearly understood and 
agreed to by both parties. 

2- The supervisory officer commits himself to provide consultative or other 
supportive assistance. 

3. Each party retains a copy of the statements. 
MONITORING 

1. Supervisory officer provides the assistance as agreed to, 

Nov. 

March 2. Original objectives are revised if necessary. 

3. Data is collected by principal and supervisory officer. 
COOPERATIVE REVIEW 

1. The principal and supervisory officer meet to: 
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Figure 7 (continued) 



April -review and assess progress toward the achievement of the objectives in the 

May statements, 

June 

-discuss the general performance of the principal, 
-discuss possible objectives for the next year. 

2. The supervisory officer prepares the Principal Performance Review Summary. 
Additional comments should focus on the principal's other contributions to the 
school. If some aspects of the principal's performance need improvement, suitable 
objectives should be incorporated into the mouel. 

3. The principal completes, if he so desires, the Principal's Comments section on 
the Principal Performance Review Summary and a copy of the signed, completed summary 
IS returned to the supervisory officer. A conference would be held on request. 



First, objective-setting is a collaborative process, with the principal and his or her 
supervisor together determining feasible objectives. Among the 47 boards with an 
objective-setting component, 24 also specify that the objective-setting process is 
collaborative, while a much smaller number (six) suggest that the principal sets the school's 
objectives without such collaboration. A small number also involve school staff or community 
in setting objectives to some extent, as is the case in the board from whose policy figure 7 
is drawn. 



Second, objectives at the principal or school level are often set within the general 
framework of the objectives set for the entire school system (and in this case, too, within 
the framework of the superintendency). Fifteen of the boards were explicit in stating *his. 

Third, objective-setting is an explicitly cyclical process, with unmet objectives from 
one cycle being considered in the specification of objectives in the next cycle. Twenty-two 
boards made this point explicit in their principal policies. Usually, objectives are set on 
an annual cycle, and in most cases this process takes place whecher or not there is a for.nal 
appraisal taking place that year. That is, objective-setting typically is done anyway, and in 
some years it may be used partly for the purpose of appraising the principal. 

Fourth, objectives are modifiable during the course of the year. Often the mechanism for 
doing so is through a scheduled mid-year meeting to monitor progress and to negotiate, if 
necessary, new object'* v/es. 




In summary, objective-setting is much more common for principals than for teachers. In 
addition, for principals it is a long-term, ongoing process. There is nothing analogous to 
the process whereby the teacher sets lesson objectives and is assessed on the basis of this 
short-term activity 

D^ta Col lection 

Methods 

There were several methods reported as being used in collecting data for appraising 
principals, as is evident in table 23. Interviewing was used most often (39 boards), followed 
by observation (18 beards), and examination of documents (11 boards). 

Sources of Information 

In a pattern quite different from that for teacher appraisal, information about principal 
performance was collected not only from the appraisee but also from other school staff members 
and, to some extent, from superintendents, parents, and students. 

Teachers were named as sources of information approximately twice as often as any other 
group apart from the appraisee. In most cases, they were to be consulted fairly informally 
regarding their principal's performance, but one board's policy included a detailed 
questionnaire about the principal which was sent to all teachers in a school, and another 
board was ver> specific about the purpose and degree of teacher involvement in the principal 
appraisal process: 

c) Initial meeting with School Staff 

Prior to the Review the Superintendent of School meets with school staffs 
of Principals receiving a Performance Review. 

Purpose: 

(1) To establish a positive attitude toward the evaluation. 

(2) To review the process of the evaluation. 

(3) To clarify to the teachers their role in the Process. The 
Superintendent of Schools must make it clear that teachers are not 
beino evaluat'id and that no writter reports on individual teachers 
wi 1 1 be prepared 
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The Performance Review Team will visit classrooms and arrange interview 
times with some of the teachers. 

In most elementary schools all teachers will be visited In all large 
elementary schools the Team may select some but not all staff members to 
visit. 

In secondary schools all department heads will be visited and a sampling 
of teachers will be seen. 

No written reports will be prepared on teachers. 

The purpose of these visits is to allow the Performance Review Team to 
compare the observations of the Principal with the operation, practices 
and procedures of the school.... 

h) Second Meeting with School Staff 

Superintendent of Schools will meet with the school staff on the last day 
of the visit. 

Purpose: 

(1) To enhance the positive attitude toward the evaluation. 

(2) To communicate in general terms the impression about the school 
received by the Performance Review Team. 

(3) To enhance the understanding that the Principal, not the staff, was 
evaluated. 

(4) To continue to develop rapport between the Administration and the 
teaching staff. 

Note: It is understood that specific observations and recommendations made 
to the Principal regarding his/her performance would not be a part of 
this report to the staff. 

The only group, other than teachers, to which extensive reference as a source of 
information was made was parents. In three of the five boards which mentioned parents, formal 
surveys regarding the principal were to be administered to parents. All five ooards making 
explicit reference to parent input were separate school boards. 



Appraiser{s) 

A principal's own superintendent is most often named as the appraiser, and the appraiser's 
Identity is one of the most commonly specified components of principal policies. 
Self-appraisal, appraisal by directo. , and appraisal by superintendents other than the 
principal's usual one are also mentioned by a substantial number of boards. Table 24 
describes the circumstances for involvement of these groups. Self-appraisal is in almost all 
cases conducted as part of the preparation for the "real" appraisal, and is used in this way 
in a somewhat higher percentage of boards (34.9 per cent of those with principal policies) 
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than is the case for teachers (23. b per cent or poaras witn ceuLner ^jumk.!^:,/. tw,,^,, v,.w 
director and "other superintendents*' are involved, they serve, as does "principal's own 
superintendent", as primary evaluator. 



Table 24: Number and Percentage of School Boards Specifying Particular Primary and 
Contributing Evaluators in Principal Appraisal Policies 



Component Number of Percentage of Percentage of 

Boards All 119 63 Boards 

Responding with Principal 

Boards Policies 

Primary evaluators 

appraisee 2 1-7 3-2 
appraisee ' s 

superintendent 46 38»7 7^-0 
other 

superintendents 11 9-2 17-5 

director 9 7.6 14-3 

others 5 4.2 7-9 
Contributing evaluator(s) 

appraisee 20 16.8 31 -7 

director 1 0.8 1 -6 

others 1 0.8 1 .6 



A closer examination of policies suggests that the involvement of other superintendents 
is almost always as members of the evaluat^'on team reported as part of policy by 13 boards. 
In three of the boards reoorting the director as primary evaluator, the director's involvement 
was as a member of the evaluation team. The other boards in which the director was involved 
as an evaluator were, in most cases, very small ones. 

Evaluation Teams 

The use of evaluation teams was one of the features of principal appraisal that was mc;t 
distinct from teacher appraisal. 

Identified by 13 boards as the prescribed method for principal appraisal, the evaluation 
team approach typically involved superintendents as evaluators, pre- and post-conferences 
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between team and principal, the entry of the team into the school for a prescribed period, and 
the principal's preparation for the team of a particular package of materials. Though these 
features were shared, the detail in which the process was specified varied from board to 
board. The following excerpt provides a brief outline of the process: 

5. An evaluation team will consist of not fewer than two Board of 
Education supervisory officers, one of whom will be the appropriate 
assistant superintendent for that school, '^oard employees, other than the 
supervisory officers, may be added to the evaluation team by mutual 
agreement of the principal and supervisory officers. 

6. The length of the evaluation process will vary from school to school but 
will normally occur over a period of not fewer than three days which mav 
or may not be consecutive. 

7. The principal should expect the following process to be used by the 
evaluation team during the in-school evaluation 

(a) Discussion on the first day of any or all of the items outlined in 
the Role Expectations of . . . Principals. 

(b) Examination of any records, communication documents, program outlines 
and/or courses of study within the school. 

(c) Observation of and discussion about the learning environment. 

8. (a) The team will meet with the Principal on the last day of the visits 

to review and discuss observations and perceptions which will be 
outlined in the written report. 

(b) A written report prepared by the team will be delivered to the 
Principal by the appropriate Assistant Superintendent within two 
weeks of the visit. 

Another board's policy, including the same features but outlining the evaluation team process 
in considerably more detail, is shown in figure 8. 



Figure 8: Role of Evaluation Team in Principal Appraisal (Policy Excerpts 



Procedure 

1. The Director shall designate a four member evaluation team and a Chairman of 
the evaluation team. 

(i) One member of the evaluation team should be the superintendent in charge 
of that school but he should not act as chairman. 

(ii) One member should come from the business area. 

2. At least two weeks before the evaluation date, the principal will be asked to 
submit the following written material to the evaluation team on a form to be 
provided. 

(i) A written statement of the goals and objectives for the school, which has 
been developed in consultation with staff. This should comprise a summary 
of the major thrusts of the school for the current year and a statement of 
general aims and philosophy. 

63 ^ 



Figure 8 (continued) 

(ii) The principal should state in written form in a general way the methods of 
staff evaluation used in the school. 

(iii) A statement of program offerings should be made available. In the case of 
a secondary school the course brochure for the current year will suffice. 

(iv) A staff list should be provided. 

(v) A school map should be provided. 

(vi) A general statement should be provided of policies related to student 
placement, promotion, p'^ovision for individual di fferences, options 
available! electives, house systems, etc. This need not be a duplication 
of the course brochure, if such information is avai able therein. 

(vii) A general statement regarding professional development of the staff 
including a brief description of time, frequency, agenda for staff 
meetings and a description of in-service activities during the past year 
and plans for the current year. 

(viii) A self-evaluation of the work of the principal on the form provided. 

3. In addition to the written statements referred to above, the principal should 
be prepared to discuss the following with the evaluation team: 

(i) staff evaluation for the previous two years 

(ii) timetabling 

(iii) pupil record systems 

(iv) procedure for reporting to parents 
(v) use of human resources — staffing, teaching and non-teaching use of 
volunteers 

(vi) counselling services 

(vii) student organizations 

(viii) extra-curricular programs 

(ix) office routines and handling of funds 
(x) communication with feeder schools 
(xi) community relations 
(xii) provision of pupil supervision re lunches, buses, field trips, etc. 
(xiii^ condition of school plant, facilities available, future plans for 

updating, renovation, maintenance, concerns of the principal regaromg the 
building 

(xiv) budget preparation and control 
(xv) school and classroom management and discipline 

The principal may report on any of these items in writing, if he chooses to do 
so. It is suggested that the team meet with the principal to discuss and 
clarify his submissions at the outset of the visit. 

4. The evaluation will take place within a one week period if possible and will 
vary according to the size ot the school and unforeseen commitments on the part 
of the team. 

5. The post evaluation assessment will include the following: 

(i) immediate verbal feedback to the principal within two or three days 
through the Superintendent. 

(ii) a meeting by the team with the principal. 

(iii) a follow-up meeting with staff by the evaluation team. 

(iv) a written report within two weeks of the completion of the visit. 

6 The report will be prepared by tne Evaluation Team in consultation and signed 
by the evaluators. The Superintendent will present the report to the principal 
for discussion and signature. One copy of the report will be retained by the 
principal and one copy filed in the Director's office. 
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Figure 8 (continued) 
Team Process 

1. Review material prepared by the principal. 

2. Meet with the principal to discuss material presented by him and to explain the 
areas of concentration of each member of the evaluation team. 

3. Meet with staff to explain evaluation procedure and to survey the staff. 

4. Mid-week meeting to discuss concerns and progress of the evaluation. 

5. At week end each member should submit a written evaluation of his area, 
including recommendations if any. 

6. Chairman of evaluation team should consolidate report and meet with team to 
discuss final »^nort. 

7. The team should meet with the principal and then with staff to inform them of 
the results. 

8. A copy of the report should be filed with the Director. The Director and 
designated superintendent should review recommendations with principal within 
six months. 



Time Spent 

Comparison of table 19 with table 23 indicates that the frequency and duration of appraisals 
are much less often a component of principal policies than of teacher policies, with only 27 
boards specifying frequency and on^y nine including information on the required number of days 
for collecting data in the school as part of an appraisal. 

More detailed information about the precise frequencies for principal appraisal 
prescribed by board policies appears in table 25. Among those boards specifying frequency, 
the most common requirement is for annual appraisal. From table 26, we see that among the 
small number of boards that explicitly consider the length of the appraisal process, roughly 
half prescribe a particular number of days and half specify that appraisal visits be 
intermittent and/or spr<3c,d over a parioo of time. 
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Table 25: Numbet and Percentage of Boards Specifying Particular Frequencies of Appraisal 
for H.M'ncipals 



Frequency Number of Percentage of Percentage of 



Boards All 119 63 Boards 

Responding with Principal 

Boards Policies 

Annually 11 9-2 17.5 

Every 2 yeai3 2 1.7 '5*2 

Every 5 years 6 b*0 9.5 

Every 4 years 6 5.0 9.5 

Every 5 years 2 1.7 3«2 



Tabl e 26 Number and Percentage of Boards Specifying Particular Number of Days per Appraisal 
for Principals 



Number Number of 

of Days Boards 
Per Appraisal 



1 day 
5 days 
p days 

intermittent, 
over a period 
of time 



Percentage of Percentage of 

All 119 63 Boards 

Responding with Principal 

Boards Policies 



0.8 
0.8 
2.5 

3.4 



1 -6 
1 .6 
4.8 

6.4 



Reporting and Follo '-Up 



Post-evaluation Conference 



Arrong the boards providing principal policy information, 52.4 per cent require a 
post-conference between the prircipal and the appraiser, making it a less frequently specified 
component than for teachers, fh.s is in contrast to the situation for pre-conferences , where 
these are more frequently specified for principals than for teachers. 
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Form of Report 



Thirty-four boards, in their policy documents, included some specification of the form of the 
report resulting from principal appraisal; the precise pattern of required report^'ng is 
illustrated in table 23. 

Most frequently, appraisers were free to select the areas of principal strength and 
weakness rn wh^'ch they wished to comment. When boards were coded as requiring an 
"unstructured statement", the appraiser had total discretion; w^en they were identified as 
requi.ing a "statement under several headings", the appraisers had to make their remarks in 
relation to a particular set of areas, often corresponding to the board's criterion domains. 
For one board, for example, these were: Leadership, Administrative, Community, Curriculum, 
Personnel , and System. 

Substantially fewer boards specified that rf»pcrts resulting from principal appraisal 
involve ratings, either of overall performance or of component behaviours. 

Overall, in contrast to the situation for teachers, the form of report was a relatively 
infrequently specified component of principal appraisal. 

Destination of Report 

Similarly, the destination of the report was specified for principals less frequently than it 
was for teachers. When it was specified, most boards indicated that copies of principal 
reports should go to the appraisee and to the board office. 

Fol low-Up 

Policies of 20 boards made some reference to planning for growth or improvement as a 
consequence of the principal appraisal process. Of these, only four provided any elaboration 
of how such planning might be conducted. Only 12 boards mentioned that there would bo some 
sort of monitoring of the plan for asiistance. 
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Excerpts from th^ee of che policies with the greatest specification regarding planning 
are presented below. 



In the first case, planning for improvement occurs at the very end of the appraisal 
process and is suggested only for principals identified as "fair" or "weak". For the fair 
principal . 



A written Plan for Improvement is to be created by this individual nnd the 
evaluator and submitted with the evaluation report. Actions taken to effect 
improvemert will be monitored by the evaluator within the time pe^^icd 
indicated. A second evaluation within the next five year period ma^ be 
required. 

For the weak one: 

A written Plan for Improvement is to be created by this individual and the 
evaluator and submitted with the evaluation report. Actions taken to effect 
improvement will be monitored by the evaluator within the time pe-^iod 
indicated. A second evaluation is required after a reasonable time has been 
allowed for improvement. 

The second case comes from a board which applies the same appraisal model to all 
categories of staff. In this board, planning is not a follow-up to a report on the appraisal. 
In it, the evaluatee is expected to plan and seek out the resources to implement plans 
somewhat earlier in the evaluation cycle. An excerpt from unis board's policy appears in 
figure 9. 



Figure 9: Planning for Principal Improvement Following Appraisal (Policy Excerpt) 
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Establ ish Goals 

(a) Co-operatively developed based on evaluation of current performance. 

(b) Goals must be: 
-chal lenging 
-attainable 

-realistic in nature and number (3 to 5 max.) 
-stated in c^^ar, accurate language 
-observable 
-measurable. 

Develop Plan of Action 

(a) Outline responsibilities of identified participants in respect to the process. 

(b) Discuss anticipated results. 

(c) Identify available assistance, support, and resources necessary to accomplish the 
goals Csee #7). 
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Figure 9 (continued) 

(d) Agree on format for collecting and recording of information. 

(e) Agree on time intervals for monitoring results and progress. 

5. Monitor and Evaluate 

(a) Discuss progress. 

(b) Initiate self evaluation by pa. ticipants. 

(c) Consider feedback from ether sources. 

(d) Discuss results and evaluate prcgrass. 

6. Affirm or Modify Goals 

(a) Affirm goals; modify if necessary and identify difficulties. 

(b) Goals are not to be confused with activities and all participants must be ^ware that 
there is a wide variety of goals (improvement, maintenance, enricnment, personal). 

7. Plan for Assistance 

Intensive assista.ice for maintaining goals through: 

-special courses 

-conferences 

-visitations 

-inservice (on tf^e job) tra-lning 
-professional counselling 
-further education 

-special assistance/additional feedback on a regular basis 

-career planning 

-exchanges 

-sabbaticals, leaves (short and long term). 

8. Summarize Evaluation 

Co-operative evaluation of previously specified goals which should include: 

-statement of agreed upon goals 
-summary of initial review 
-assistance provided 

-a written report of areas of agreement and disagreement 
-recommendations for future action. 



In the third example, the exact timing for seeking professional assistance is not 
specified. Instead, the resource list presented in figure 10 is included as part of the 
pol icy manual . 



^^'g^^^ ^Q- Resources f or Professional Assistance for Princip al Improvement Followinq 



Appraisal (Policy Excerpt) 




RE SOURCES FOR PROFESSIONAL ASSISIANCE 

In emphasizing the need for continuing professional growth and development, this section 
outlines concrete examples of assistance which is available. While the examples listed are 
not exhaustive, they offer some guidance for individuals with a desire to improve and grow as 
a professional. While assistance may be recommended by the review team, the principal has a 
responsibility as a professional, to take advantage of available opportunities for 
professional growth. 
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Figure 10 (continued) 



Opportuni ties for Financial Assistance/Growth 

1. Board Professional Development Funds. 

2. Board Conference Funds. 

3. Federation Funds for Professional Development and Conferences. 

4. Ministry of Education Funds for Professional Development. 

5. Funds available through the Board/Branch Affiliate Collective Agreement. 

6. exchange programmes within the Board, the province and interprovincial and international 
exchanges. 

7. Transfer within the jurisdiction of the Board. 

8. Support to attend retreats, live-ins, etc. 

Opportunities for Assistance 

1. Professional Literature 

Professional readings from books, articles and magazines is a valuable source of new and old 
information on various educational topics. These readings may come from the individual's 
personal library, a colleague, the school, the Board Professional Library or through libraries 
to which the Board has accp*;". at various Faculties of Education. 

2. Research Services 

Individuals can make use of computer-assisted research services to which the board subscribes 
as a source of literature and materials. 

3. Visitation 

Arrangements can be made for individuals to visit the schools of successful principals either 
within the school system or externally. 

4. Workshops, Seminars, Conferences 

Where an area for growth and/or improvement has been identified, this individual can seek 
assistance through attendance at a presentation in the area identified. 

Consultative Assistance 

Depending upon the area for growth and the availability of personnel, arrangements can be made 
for consultative assistance through Board Personnel, Colleges of Education, Federation 
Personnel and the Ministry of Education. 

6. Courses 

Individuals with a desire/need for professional growth can undertake studies through Ministry 
and University courses which are available on a year-round basis. 

Figure 10 (continued) 

7. Self-Evaluation 

Various instruments are available for individual use which assist t..e principal in analyzing 
his/her effectiveness. These are available through the Board Office and can be employed 
co-operat 'vely or on an individual, private basis. 

8. Exchanges. Educational Leaves 

For the experienced principal who desires further professional growth, there are a variety of 
exchange plans available. Information on these is available through the Board Office or the 
Branch Affiliate. Educational Leaves (Sabbaticals) both paid and unpaid, which meet the 
criteria of professional advancement and the needs of the system, are available to 
individuals. 



8/ 

70 



Differentiation of Procedures 



Different Categories of Principal 



Only seven boards differentiated among procedures to be used in appraising various categories 
of principal. In most cases, this involved minor procedural differences in the conduct of 
appraisals of elementary and secondary school principals. In addition, in one board the 
frequency of evaluation for newly appointed differed from that for more experienced 
principals; in another, there were differences in procedures for those with continuous as 
opposed to term appointments. 



Satisfactory and Unsatisfactory Principals 



Only three boards gave any special attention to procedures for principals found to be 
performing unsatisfactorily. This sHuation is in marked contrast to that for teachers, where 
42 boards outlined, some in extensive detail, the procedures to be followed for teachers 
considered to be performing at a less than acceptable standard. Of the three boards whose 
principal policies considered the natter, only one had more than a few sentences. 



Explicit Differentiation of Aporaisal Systems into Two Types 



Although a few boards also made brief references to some sort of differentiation in the 
purposes or focus of their principal appraisal system, only five boards made explicit the 
nature of this differentiation. This situation again stands in marked contrast to that for 
teachers, where 19 boards differentiated their evaluation systems, using formati ve/summati ve, 
administrative/non-administrative, supervision/evaluation, or classroom/comprehensive 
distinctions. 



For their principal appraisal systems, two boards usea the formati ve/summati ve 
distinction, which was the most common distinction among teacher evaluation rystems and which 
is illustrated by the following example: 



Formative Review 

Formative review occurs during the process of managing and administering a school 
and is a supportive, d.-^velopmentai process directed at the following goals: 
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-To encourage and support the practices and qualities of the successful Catholic 
Principal . 

-To identify the needs of the individual principal for continued professional growth 
and development. 

-To provide a means for identifying, maintaining and improving the quality of 
education in our school's. 

Summative Review 

The summati'e review of a principal judges the effectiveness of the management and 
administration of a school by an individual in order to meet the following goals 

-To determine if continued appointment will be recommended. 
-As a basis for the demotion of an incompetent practitioner. 
-To determ-''ne if an increment will be granted or withheld. 
-As a basis for the promotion of an individual to a position of greater 
responsibi 1 ity. 

The other three boards differentiated the appraisal process essentially into a less formal and 
more frequent process and a more formal and less frequent process. In all three boards the 
former orocess was characterized by annual objective-setting under the supervision of the 
superintendent. The latter process typically included, in addition to the same 
objective-setting, more frequent contacts b<?tween principals and their appraisers, and more 
detailed reporting on the appraisal. Figure 11 illustrates the difference between the two 
processes. 

Figure 11: Differentiation of Principal Appraisal System into Two Types (Policy Excerpt) 



1. OVERVIEW.... 

Application of All Principals will receive an annual evaluation, either formal or 
Evaluation Model informal. 

Types of A: FORMAL EVALUATION 

Evaluation 

Formal evaluation of each Principal will occur every four years. Annually 
each Superintendent will select the appropriate number of Principals to be 
evaluated by him and will advise those individuals accordingly following 
placement of Principals fo*- the forthcoming year. During the year of 
fo.mal evaluation, the Superintendent will be required to visit the school 
in the fall term to establish strategies for the evaluation; to 
subrequently visit the school through the schoo^ year a minimi"^ of five 
times for the purposes of evaluation; and to submit to the Di .-Ctor of 
Education a formal pvaluation reoort at the conclusion of the evaluation. 

B: INFORMAL EVALUATION 

Informal evaluation of each Principal will occur in each year excepting 
the year in which formal evaluation is scheduled to ^-ake place. At the 
commencement of each school year, the Principal will determine the aims 
and objectives for the school. The sdiool Superintendent will visit tlie 
school in the fall term to discuss the statement of intent with the 
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Figure 11 (continued) 



Evaluation Cyc^e 



Evaluation 
Reports 



Principal. In the second term, a return visit will be made by the 
Superintendent to discuss with the Principal progress with respect to that 
statement of intent. 

Following this discussion, the Principal will submit to his Superintendent 
a written summary of progress toward the achievement of the aims and 
objectives for the school for that year. 

The annual evaluation cycle for both formal and informal evaluation begins 
with the identification by the Principal of his aims and objectives for 
the school year. A plan of action is then established to determine how 
these are to be achieved, and to agree on activities in which Principal 
and Superintendent are to be involve-'. Following the implementation of 
the plan, in the year of formal evaluation, evaluator and evaluatee 
participate in an interim review and make adjustments where required to 
the initial plan of action. 

Where formal evaluation is scheduled, the submission of the evaluation 
report Dy the Superintendent to the Principal is the next step in the 
cycle, and is followed cy a post-evaluation conference between the two 
individuals. The repo'^t is then submitted to the office of the Director, 
and the cycle is completed by the important follow-up to evaluation which 
includes the implementation of the recommendations contained in the report 
and their possible incorporation into the aims and objectives for the 
following school year. In this • .y, the cycle builds on previous 
experiences and pro .tes the improvement and growth which are basic 
purposes of the evaluation process. 

In those years where informal evaluation occurs, a year-end summary report 
of progress will be forwarded to the Superintendent by the Principal. 
While flexibility is to be allowed with respect to the time for the 
submission of the report, the latter must occur prior to the completion of 
the school year. . . . 



THE SUPERINTENDENT'S FORMAL REPORT 



Purpose of the 
Report 



Local Conditions 



i) The primary purpose of the Superintendent's Formal Report is to act as 
a guide in measuring the continuing development of the school, principal, 
and staff. It is a reporting instrument which provides the Superintendent 
of Education with a formal means tor recording his monitoring and 
assessment of the direction, programmes, organization , supervision, 
management, staff development and Catholic leadership within the school. 
The Report represents a culmination of the Superintendent's dialogue with 
the Principal, discussions with the staff and students, and involvement in 
the school, as well as his observations and visitations throughout the 
school year. It, therefore, provides a regular review, and servcis as a 
mechanism for furnishing a profile of strengths and weaknesses which forms 
the basis for the initiation of further growth and development. 

ii) The Report shall take into account ,ocal school and neighbourhood 
conditions constraints which might influence significantly the 
educational programmes and processes in the school. Such factors might 
incUde the following: 

-the character or make-up of the school population (e.g. dominance of 
learning problems, affluent families); 

-the type of araa or neighbourhood served by the school (e.g. inner city, 
new expanding subdivisions, socio-economic structure, ethnic influence, 
decline or growth area); 

-staff make-up (attitudes, experience, turnover); 

-notable physical plant limitations and conditions (e.g. absence of 
library or gym facilities, substandard classrooms, portables); 
-other extenuating circumstances (e.g. disruptive renovations programme 
for six months of the year). 
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Figure 11 f continued) 

Nature and iii) The Report shall be formative in nature, anecdotal in form, and 

Contents shall be positive and constructive in character. Further, the Report 

contents shall 

-indicate the degree of progress toward defined objectives; 

-be guided by functional areas outlined in the role definition for 

Principals in this system; 

-identify accomplishments and results; 

-note strengths and weaknesses; 

-include specific examples and data, where appropriate, to support 
impressions and conc'^usions ; 

-encompass specific recommendations: (a) to the Principal and staff for 
improvement; (b) to the Director and Board to alleviate any undesirable 
situations or conditions.... 

4. EVALUATION PROCEDURES " INFORMAL EVALUATION 

Principal's vi) The Principal shall submit to his Superintendent by the end of the 

Year End school year a summary of progress towards the achievement of the school 

Summary aims and objectives. 



Evolution of Policy 



Development 



Eleven boards included, as a component of their principal appraisal policies, a description as 
to how those policies were developed. As was the case for teacher policies, the descriptive 
information consisted mainly of the composition of the committee developing policy and, in 
some cases, the purposes, history, and activities of the committee. The importance attached 
by many boards to the involvement of the group being appraised is illustrated by the following 
policy excerpt. 

In September of 1975, an intensive two-year professional development project, 
"A Study of the Principalship'* , was initiated in the ... system by principals 
for principals. Identified as part of the rationale for this undertaking was 
the need for continued evaluation of the principalship. . . . 

During the study, principals recognized that evaluation must be a cooperative 
endeavour between evaluator and evaluatee, and that those affected by the 
process should be involved in its development and implementation, r.erein lies 
a strength of the moael for Principal Evaluation which is described in this 
handbook. 

This model for Principal Evaluation is truly the result of a cooperative 
effort. Throughout the development of the model, opportunities were designed 
for principal and supervisory officer input and reaction. From the 
establishment of the rationale to the determination of the various components 
of the model, principals and senior officials were active participants. This 
same spirit of cooperation is reflected in the team approach which is inherent 
in the determined evaluation process. 
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Implementation 



Table 23 shows that only seven boards provided any information about how their principal 
appraisal system would be implemented. In general, implementation was to be carried out by 
providing in-service programs to enable development of appraisal skills. Only one board paid 
extensive attention to the implementation process. This board, when adopting a new policy, 
established an implementation steering committee, identified for the committee a series of 
implementation tasks and possible obstacles to their accomplishments, and set out a series of 
guidelines for implementation. 

Review 

Eight of the boards submitting information made sone reference to a m^'chanism for reviewing 
the appraisal procedure and, on the whole, this information was even less complete than that 
for policy development and implementation. These eight boards, in general, indicated only 
that there would be a review of the procedure; a lew provided specific detail as to how often 
the review would be conducted and/or by whom. 

Cluster Analysis 

Cluster analysis again was used to see whether or not principal appraisal systeris could be 
grouped into distinct types. Eighteen components of the appraisal process were included in 
the analysis and again no clear groupings emerged. 

However, there was a tendency for a certain set of components to be included in most of 
the policies where an evaluation team approach was used. Boards with such policies usually 
were the only ones to include specifications that particular documents ond written submissions 
should be examined as part of the appro sal, that observation should also be used as a method 
for collecting data, and that information about the principal shoulJ be collected 
systematically from some members of t... school staff. Boards with evaluation teams also 
included in their policies othor leatures (e.g., pre-conf erences , post-conferences, 
specification of the length of the appraisal) which were not unique to policies specifying 
teams. 



The fact that no clear clusters emerged around the evaluation team approach (despite a 
certain set of commonly held component features) seems again to suggest that policies are 
aggregates of independent components with different parts being based on different sets of 
assumptions. 

Superintendent Appraisal Policies 

Only 46 of the 119 boards that provided appraisal information submitted some sort of 
information about superintendent policies. Various components that may be part of a 
superintendent appraisal system '?re listed in table 27. As in table 19 for teachers and table 
23 for principals, the number and percentage of boards whose systems include these components 
are given. Percentages are expressed with respect to all 119 responding boards and to the 46 
boards which submitted superintendent information. The hierarchical organization of the table 
resembles that for tables 19 and 23, although there are some additional and some delated 
categories. 

Table 27: Number and Percentage of Boards Speci.yinq Particular Components in Superintendent 
Appraisal Pol icies 



Component Nuaber of Percentage of Percentage of 

Eoaraa All 119 46 Boards with 

Responding Superintendent 

Boards Policies 



Preparation 

Purposes 

development/ 
improvement : 

general 10 6*4 

development/ 
improvement : 

specific 12 10.1 

administrative: 

general 4*2 
administrative : 

apecific 7 5.9 



21 .7 

26.1 
10.9 
15.2 



Criterion domains 
aduiinistrative 

performance 9 7*6 19.6 
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Table 27 (continued) 



Component 



Number of 
Boards 



Percentage of 
All 119 
Responding 
Boards 



Percentage of 
46 Boards with 
Superintendent 
Policies 



school and 
community 
rex ions 

program 
organization 

personnel 
management 

contribution 
to board 



15 



Standards 
Job 

description 

Activities 
prior to 
appraisal 

request for 
statement about 
objectives , 
activities and 
plans 

request for 
self- 
evaluation 

pre- 

conference 

interim 
conference 



10 

6 
0 

15 



Objective- 
setting 

Data Collection 

Data collection 
methods 

examination 
of documents 



23 
19 
39 



12.6 

6.7 

8.4 

5.0 
0.0 

12.6 



2.5 

1.7 

19.3 
16.0 
32.8 



32.6 
17.4 

13^0 
0.0 

32.6 



6.5 

4.3 

50.0 
41.3 
84.8 



3.4 



8.7 
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e 27 (continued) 



Component Number of Percentage of Percentage of 

Boards All 119 46 Boards with 

Responding Superintendent 

Boards Policies 



interview 26 21.8 56.5 

:ther 2 1.7 4-3 

Souries of 
information 

superintendent 
beir^ 

appraised 23 1?.3 50.0 

airector 1 0.8 2.2 

board staff 1 O.S 2.2 

trustees 1 0.8 2.2 

Appraiser(s) 

superintendent 
being 

appraised 13 10.9 28.3 

other 
super- 
intendents 5 4.2 10.9 

director 35 29-4 76.1 

trustees 5 4.2 10.9 

team 2 1*7 4*3 

Time spent 

frequency 28 23-5 60.; 

Reporting 
and 

follow-up 

Post- 

conference 22 18.5 47.8 

Written 

report 18 15-1 39-1 



So 
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Table 27 (continued) 



Component 



Number of 
Boards 



Percentage of 
All 119 
Responding 
Boards 



Percentage of 
46 Boards with 
Super? ntendent 
Policies 



Destination 
of report 

superintendent 1 1 

central office 
file 6 

trustees 6 

Folio v-up 

plan 

developed 6 

monitoring 

of plan 5 

Differentiation 
of procedures 

Differentiation 
between procedures 
for satisfactory 
and unsatisfactory 
superintendents 2 

Explicit 
differentiation 
of appraisal 
system 

into two types 2 

Evolution of 
policy 

Development 

of policy 8 

Implementation 
• of policy 4 

Review of 

policy 2 



9.2 

5.0 
5.0 

5.0 
4.2 



1.7 



1.7 



6.7 
3.4 
1.7 



23.9 

13.0 
13.0 

13.0 
10.9 



4.3 



4.3 



1-^.4 



8.7 



4.3 



Preparation 



Purposes 



As was the case for teacher and principal policies, those for superintendents reveal a greater 
concern for developmental/improvement purposes than for administrative purposes. 



Among the boards providing superintendent information, the percentage including 3ome 
specification of purposes was substantially lower than for the boards submitting principal 
information. For example, for superintendents 26.1 per cent described specific 
developmental/improvement purposes and 13.0 per cent described specific administrative 
purposes, while for principals the figures were 47.6 per cent and 27.0 per cent respectively. 



Not only are the number and percentage of boards describing specific purposes for 
superintendent appraisal smaller than those for the other groups considered, but also the 
portions of these policies which describe the purposes of superintendent appraisal are very 
brief. The following, which ^reats both developmental/improvement and admin'''strati ve 
purposes, was the most comprehensive statement analysed: 



Little or no personal and professional growth occurs without a periodic 
inventory of strengths and weaknesses. Quality control and the pursuit of 
excellence in the performance of duties are principles to which public servants 
must ascribe. The supervisory officers of the ... Board of Education have 
articulated and have accepted the following general purposes for systematic 
p'^rformance reviews: 

1. To provide answers to the question, "What ao they do?" 

2. To provide opportunities to secure validation of the roles and to give 
direction to those performing the roles. 

3. To provide the systern with methods and opportunities to analyze its 
various components so that priorities may be set, resources allocated 
appropriately, directions given and improvements made which will 
positively affect the quality of education for pupils. 

4. To p^o'ide information so that the system can better make decisions 
respecting promotion, tenure, task assignment and remuneration. 

5. To stimulate self growth and to have individuals recognized for their 
contributions. 
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More typical in their brevity are the following two: 



The Administrative Council believes that a means of maintaining and improving 
the quality of management services now provided in our jurisdiction is to 
conduct a system of annual performance review for all senior personnel Such a 
system should gain two advantages: 

(a) Each individual's performance will be more finely tuned to meet the 
Board's objectives. 

(b) Each individual's skills will be assessed objectively and brought 
directly to bear on the current administrative problems. 



Performance Review should: 

(a) help the individual grow boti. personally and professionally, 

(b) protect both the employee and the employer against unfair practices 
and ' 

(c) result in a general improvement of the system. 

One stated purpose of appraisal that was unique to the superintendent level was the use 
of performance appraisal results to influence, in some way, salary decisions. While this 
purpose was specified by only four boards or 15.2 per cent of the 46 responding, these four 
constituted more than half of the seven considered in table 27 to have had "administrative: 
speci^-'c" components. How the appraisal information wa5 used in salary decisions was not 
particularly detailed. 



For example, from a board document: 



I 



Through this process, rewards for performar.ce can be fairly given. For some 
the rewards may be monetary. Apart from salary, other awards include honest 
feedback, recognition of good performance, support for training needs and so 
on. 

An additional example comes from an informal letter submitted instead of a formal policy 
document: 



Our salary ranges are determined by the HAY Salary Determination Plan and our 
individual placement within the range is based upon performance. (This is 
accountabi 1 ity! ) 
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Criteria, Standards, and Job Descriptions 



ERIC 



From table 27, we can see that, among the 46 boards providing superintendent information, 
there is no evidence of rigorously defined standards of expected performance, apart from those 
that may be set in an objectives-based appraisal system. In addition, in contrast to the 
situation for teachers and for principals, there is relatively little attention paid to the 
specification of criteria. However, a component rarely seen in teacher and principal 
policies, namely, an explicit reference to a job description, is apparent in a substantial 
number of superintendent policies. 

In analysing such policies from boards which did consider the matter of criteria in some 
way, we found that the criterion domain notion, so useful in the analysis of teacher and 
principal policies, was well suited to the superintendent policies of only a limited number of 
boards. Although seven boards did outline expected superintendent behaviours in terms of 
criterion domains such as administration, program, personnel, and community/board, there was 
difficulty in applying the notion of criterion domain to the superintendent policies of some 
of the other boards. Several factors account for this situation. 

First, several boards outlined expected performance for superintendents in terms of 
general management skills (or competencies, or, in one case, management factors) and/or 
qualities, rather than outlining specific expected behaviours in various domains. There was 
considerable variety among these boards in the amount of detail specified. Typical of the 
less complex was the following: 



The duties of the Senior Academic Officials include but are not limited to 
planning, directing, controlling, analyzing, communicating and organizing. An 
individual's effectiveness in the performance of these responsibilities will be 
reflected in the results achieved and described through a goal /objective- 
setting process. 

At the same time it is reccjnized that inter-personi.1 effectiveness and an 
individual's ability to promote good working relationships within and without 
the school system are vital to successful support of the Board and its aims, 
objectives and pclicies. 

A more elaborate example, which is characterized to some extent by the management skills 
approach, appears in figure 12. Both examples also reflect the central ity of the results of 
obj'^cti ve-setting in assessing performance. 
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Figure 12: Criteria for Superintendent Appraisal (Policy Excerpt) 



FACTOR 

1. MANAGING STAFF 

Cooperation and teamwork 
Exchange of ideas 
Building esprit de corps 
Trust and respect of staff 

2. KNOWLEDGE OF JOB 
Scholarship 

Thorough knowledge of job 

Understanding interrelationships with other functions 
Objectivity 

Keeping abreast of new developments 

3. EMPHASIZING GOALS AND ACHIEVEMENT 

Clarification of group objectives 
Encourage employees 
Carry out system ^oals 

Set and maintain high employee performance standards 

4. PROBLEM SOLVING AND DECISION MAKING 

Integration of information 
Identification of problems 

Separation of non-essential from critical data 
Creative and innovative ideas 
Analytical thinking 
Objecti vity 

5. ORGANIZING AND FACILITATING WORK 

Organization and planning 
Defining work relationships 
Delegation, control, follow-up 
Communication 
Flexibi 1 ity 

6. APPRAISAL AND DEVELOPMENT OF STAFF 

Realistic and objective appraisals 
Good record of employee performance 
Quality of reviews with staff 

Follow-up to help staff improve and set up development plans 

7. ECONOMICS MANAGEMENT 

Efficient use of finances, budgets, facilities, equipment, and supplies 



Another reason that criterion domain was somewhat less useful for the analysis of 
superintendent policies was that, in several boards, the expected behaviours for 
superintendents were elaborated in terms of duties to various groups with which the 




superintendent worked rather than in terms cf particular functional domains or types of 
activities. An example of this approach (which again makes explicit reference to the setting 
of objectives) appears in figure 13. 



Figure 13: Superintendent Responsibilities (Policy Excerpt) 
C.2 Expectations of tne Supervisory Officer 
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In addition to the Acts and Regulation, it is the expectation of the ... Board of 
Education that each supervisory officer will develop a positive learning environment in 
the school in each of the following areas: 

a) System Expectation for 19— 

I. Role in Administrative Council 

-reflects views of Family/Department to Council members, 
-accepts constructive criticism by peers at Council. 

-contributes ideas, professional expertise to Council to assist in quality decisions, 
-assists in making the Administrative Council function smoothly, effectively, 
-listens to views of other Council members. 

-prepares information, reports and recommendations to Administrative Council on time, 
in a clear manner for discussion and disposition. 

-supports Council decisions in a positive manner in Family/Department. 

-communicates Council business to Family/Department. 

II. Role as Liaison Member to Association/Federation 

-attends as many of the Association/Federation meetings as possible or is invited. 

-takes a part in the organization of meeting, topics to be placed on the agenda 
together with the executive of the Association/Federation. 

-keeps Association/Federation informed of Administrative Council, Board policy 
changes. Administrative regulations. 

-consults with Association/Federation to ascertain views on issues to come before the 
Administrative Council. 

-reflects the views of the Association/Federation at Administrative Council re issues 
of concern. 

-informs the Administrative Council of items discussed at liaison meetings, 
recommends follow-up. 

III. Executive Membership on Board Committees 

-works with the Chairman to establish the agenda, meeting time/place. 

-reviews the agenda items with the Director and any other staff involved prior to the 
meeting. 

-carries out the follow-up required and informs Administrative Council of key 
matters. 

-assists the Chairman in the operation of the meeting. 
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Figure 13 (continued) 



-keeps in close contact with the various trustee members of the committee re agenda 
items, information. 

-wo^ks with the Assistant to the Director and Recording Secretary re minutes of 
meetings, agenda, publicity, etc. 

IV Special Studies Responsibility 

-work5 within guidelines/terms of reference of study. 

-selects and acquires appropriate staff to assist in study. 

-gives all those publics concerned an opportunity to contribute in a meaningful 
mmner through the development of a clear process for the study. 

-keeps Administrative Council, Board informed of progress. 

-establishes time line and adheres to commitments for submission. 

-submits a complete report with firm recommendations for Administrative 
Counci 1 /Board. 

V. Other 

-works within the framework of the ... Philosophy of Management. 

-has the concern and needs of students, staff, associates, uppermost in mind in 
decision-making. 

-is positive in public support of actions of fellow staff, trustees. Board of 
Education. 

-seeks to improve himself as a professional educator, 
-leads in a pleasant, open, honest manner. 

-gives credit and recognizes others' contribution to the system, 
-is a team man. 

(b) Family/Department Objectives for 19-- 

The staff within your Family/Department look upon the Superintendent as their leader. 
Together with them, they expect that he will assist in the establishment of objectives 
for the Family/Department which are realistic, attainable and within those of the Board 
of Education and its resources. The process of firming up these objectives is viewed by 
staff and Director as equally important to the actual implementation and completion of 
the Family/Department objectives. Consequently, Superintendents must use their 
leadership skills to help identify problems, resources needed to solve them, and 
measurement tools required to evaluate successes/failures. He must also recognize the 
time input required by staff and temper his annual planning with this in mind. 

(c) Personal Expectations for 19-- 

Each year, the Superintendent must examine his own Family/Department and how he operates 
within the job expectations. It is a goal of all of us that we will attempt to improve 
upon our leadership role, our administrative procedures, our handling of people, our 
communications and public relations roles, etc. As Superintendent, he should identify 
these in terms of an opportunity to share them with the staff and the Director in order 
that it is possible for a judgement to be made with him as to the relative success in 
meeting these expectations. 
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Despite these difficulties, all superintendent policies and/or procedures which referred 

to criteria in any way were coded in terms of domains, and the one most frequently mentioned 
was the superintendent's need to communicate and co-operate with the community. 



Fifteen of the 46 boards made explicit reference to the position description as a means 
of determining whether or not performance was at the expected level. Of these, six boards 
submitted detailed descriptions of the responsibilities of parcicular positions Although 
these descriptions look very much like the lists of behaviours expected of a "good" 
superintendent, they are written in terms of specific responsibilities expected in a unique 
position rather than in terms of ideal performance in a superintendent's role. This explicit 
focus on particular position descriptions may constitute yet another reason that the number of 
boards outlining criteria for ideal superintendent performance is so low in contrast to that 
for ideal teacher or principal performance. 

Activities Prior to Appraisal 

Two board", specified that a superintendent should do a self-evaluation in preparation for 
appraisal, and three specified that the submission of written objectives be used to initiate 
the appraisal process. 

Twenty-three boards had as a component of their superintendent system the specification 
that a pre-appraisal conference be held. This conference in almost all cases involved the 
setting of objectives against which later performance was to be assessed and the specification 
of methods to be used in attaining objectives. 

Nineteen boards specified that, in addition to the original objective-setting conference, 
another conference be held later to review progress and, if necessary, to revise objectives 
and plans. 

Objective-Setting 

The setting of objectives is central to the appraisal of superintendents. As is seen in table 
27, 39 of 46 or 84.8 per cent of the boards reported the use of objective-setting. 
Comparisons with tables 19 and 23 show that, respectively, 42.7 per cent of boards submitting 
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ccacrier policies ana /4.b per cent of boards submitting principal policies indicated 
objective-setting components of their policies. 

The terminology and exact detail of the objective-setting process v^^y considerably among 
boards. Some, for example, concentrate objectives within "key result areas" and develop clear 
indicators for the successful achievement of objectives. Others use the "manager's letter" 
approach to objective-setting. Despite this diversity, there are, as there were for 
objective-setting for principals, several features common to many boards' policies. 

First, on the whole, objective-setting is conducted jointly by superintendent and 
appraiser. Nineteen boards of the 39 which reported the use of objectives also noted that the 
process was collaborative. However, there is a sense in the policies of some boards that, in 
contrast to objective-setting at the principal level, the process is less one of collaborative 
objective-setting than one involving delegation of responsibilities. For example: 

The senior administrators are responsible for managing the school system on 
behalf of the Board and this includes managing the achievement of the biennial 
object-* ves and long-range goals established by the Board. This policy outlines 
the processes through which these objectives will be set, how responsibilities 
will be delegated, and the schedule for reviewing achievement of the 
objectives. 

This excerpt illustrates a second feature of the process involved in superintendent 
appraisal, that objectives are set within the framework of the objectives set by the board and 
director. Ten responding boards made this explicit in their policies. 

Third, objective-setting is a cyclical process, with 19 boards reporting that they set 
objectives annually and two reporting that they do so every two years. 

In several of these boards the post-appraisal conference for one cycle and the 
pre-appraisal conference for the following cycle are one and the same. For example: 

One interview is held at least bi-annual ly for the purpose of assessing overall 
job performance for the past year, and for mutually developing and confirming 
objectives and plans for the next year. 

Fourth, objectives are modifiable. As was stated earlier, 19 boards, or approximately 
half of those reporting the use of objective-setting, also indicated that there should be a 
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meeting part way through the objective-setting cycle to review progress to date and to 
consider the revision of objectives. 



Data Collection 



Methods 



Interviewing was the method of data collection most often specified in superintendent 
policies. Four boards specified that written documents (e.g., self-evaluation forms) be 
examined as a part of the process. Only two boards supplied considerable detail regarding 
methods of collecting data. The list in the following excerpt is used by both boards: 



The process of evaluation will be based on a review of the degree of success in 
reaching objectives which have been decided on for the year and by a process of 
observation and discussion taking place over the year by the Director through: 

-visitations 

-attendance at principals* meetings 

-communication, both written and verbal 

-interaction witn fellow supervisory staff 

-interaction with trustees 

-interaction with the public 

-involvement and contribution to policy setting 

-observation of role in the committees and at Board 

-quality of work output 

-the level of acceptance of leadership 



Sources of Information 



Among boards that did specify the source of infornation in the superintendent process, the 
source most frequently cited was the individual superintendent. One board required the 
appraiser to seek information from board support personnel, while another required the 
appraiser, a superintendent, to seek information from the director and trustees. 



Appraiser(s) 



As is seen in table 27, 35 boards specified the director as the superintendent's appraiser and 
five specified that trustees be involved. Five board<;, all of which were reasonably large, 
specified that the appraiser be another, more senior superintendent. Thirteen boards 
indicated that the superintendent should be encouraged to conduct some self-evaluation. Only 
two boards had a team approach to evaluation; one had the director, and subsequently a 
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committee of the board, appraise superintendents, while the other had three trustees and the 
director meet with each superintendent to appraise performance. 

Time Spent 

Twenty-eight boards, or 60.9 per cent of the boards providing superintendent information, had 
as a component of their policies the frequency with which appraisal should occur. Of thesu, 
22 specified that appraisal should take place annually, while six specified that it should 
occur every two years. 

Reporting and Follow-Up 

Post-evaluation Conference 

Twenty-two boards specified the holding of a conference to end the appraisal process. 
Form of Report 

Eighteen boards indicated that a written report on the appraisal should be made. Only eight 
of these indicated the precise form that the report should take. While four boards specified 
the use of anecdotal reports (with comments made either in general or under particular topic 
headings), four specified the use of ratings of particular categories of behaviour. 

Destination of Report 

Eleven boards specified that a copy of the report go to the superintendent being appraised, 
in six boards, a copy of the report was to be kept in the director's or central files, and in 
another six the report was to be presented, in eUher written or oral form, to the trustees. 

Follow-Up 

Planning for improvement and monitoring of such plans were included as policy components even 
less often for superintendents than they were for principals and teachers. Only one board 
provided discussion of the subject, and an excerpt from its policy appears as figure 14. 
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Figure 14: Resources for Assistance for Superintendent Improvement Fonowlng Appraisal 
(Policy Excerpt) 



Help! 

How do I become a better supervisor? Where do I go for professional help? 

At one time, these were difficult questions to answer. It would appear that an attitude 
prevailed of having reached such a high level, competency was inherent and long term. We no\ 
know that none of us in a leadership role can afford to stand still. There will always be m 
skills to learn and new approaches to problem solving required. 

The responsibility for meeting these challenges is doth a personal and system one. In this 
section of help, we again emphasize that our suggestions are limited and i^ust be examined in 
the light of personal needs: 

a) Personal -Self Evaluation 

-Professional Reading 

-Planned interaction with peers 

-Committee leadership 

-Acceptance of responsibilities in professional organization 
-Attendance at Workshops Teacher, Principal 

b) System -Planned program of Visitation to other jurisdictions 

-Conferences 

-Workshops — OAEAO, OCLEA, C.E.A. Leadership, Ministry of Education 

-Summer Programs 

-Job Rotation/Exchanges 



Differentiation of Procedures 



Satisfactory and Unsatisfactory Superintendents 



Only two boards, as is seen in table 27, included in their policy any specification of the 
procedure to be followed if a superintendent's performance was unsatisfactory. 

Explicit Differentiation of Appraisal System into Two Types 

Only two boards made any differentiation of the appraisal system into two types. In both 
cases, there is an annual objective-setting process and in addition, every two years, a more 
detailed review of performance. To illustrate the differentiation, figure 15 contains an 
excerpt from the policy of one of these boards. This board's policy includes a very unusual 
amount of detail regarding the superintendent appraisal process. 
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igure 15: — Differentiation of Su perintendent Appraisal System into Two Types (Policy Excerpt) 



III. Yearly Evaluation 

The annual review (April - June period) will focus primarily on the success, failure or 
delay in meeting the objectives for the past year. The stages will be: 

i) The Superintendent will prepare a brief report for the Superintendent of Academic 
Affairs, outlining his/her own perceptions of what has been accomplished, can yet be 
accomplished, or will not likely be possible; 

ii) as a result of reading this report, in conjunction with the earlier statprnent of 
planning prepared jointly, the Superintendent of Academic Affairs will prepare his 
own assessment, in writing, as a basis for discussions with the Superintendent; 

iii) as a result of this discussion, areas of agreement and/or disagreement will be 
determined and the final report will be completed by the Superintendent of Academic 
Affairs for the Superintendent and the Director. 

IV. Summative Evaluations 

Normally, the first year, and alternate years thereafter, will be summative years in 
which the evaluation will be an in-depth review of the success or failure of the 
Superintendent in meeting the objectives for past years, as well as in meeting the 
expectations fot the role of Superintendent. 

The process will be similar to that of the yecirly evaluation, but shall take place at a 
mutually agr^jed upon time of the year. While based, to a great degree, on the cumulative 
record of achievement included in the yearly evaluation, the summative evaluation will 
focus as well on the suitability of the Superintendent for his role. 

Using the earlier reports of the annual assessments, the Superintendent of Acadenic 
Affairs will prepare a draft report which will also include: 

1. his perceptions and written analysis of the Superintendent's impact on the system, 
his major accomplishments, any areas of concern, his personal development plans and 
any professional (career) plans; 

2. the perceptions of the Director with respect to the items listed in #1 above; 

3. the perceptions of the Trustees with respect to the items listed in #1 above. 

Tht Superintendent of Academic Affairs will be responsible for soliciting comments in 
writing from the Director, from Trustees on the work of the Superintendent with the 
Board, and from those Trustees with whom the Superintendent has worked most closely on 
the work of the Superintendent in the community. 

Copies of the comments obtained shall be provided to the Superintendent. 

The Superintendent will also prepare his own report, (general basis) and the evaluation 
will begin with a personal exchange of written reports, immediately followed by a 
discussion to develop areas of agreement and disagreement. A final report which details 
those areas and any recommendations, and the Superintendent's comments (if any) is to be 
prepared, by the Superintendent of Academic hffairs, and communicated to the 
Sjperintendent and Director. The report is placed on file, in the office of the 
Superintendent of Academic Affairs, available for Board use upon written request of the 
... Board cf Education. Each Superintendent's personal file and evaluation records are 
available to that individual at any time. 
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Evolution of Policy 



Only eif,ht boards provided information about how lue superintendent appraisal policv was 
developed, only four provided information about implementation, and only two specified that 
the policy be reviewed. 



Director Appraisal Policies 



Thirty-eight of the 119 boards supplying appraisal information submitted information regarding 
directors. The components of their policies are presented in table 28 in a manner similar to 
that used in tables 19, 23, and 27. For each component tria number and percentage of boards 
including it are listed and percentages are expressed in terms of both all 119 responding 
boards and the 38 boards supplying director information. Hierarchical organization and 
categories are similar to tables 19, 23, and 27. 

Table 28: Number and Percentage of Boards Specif ving Particular Components in Director 
A ppraisal Policies 



Component 



Number of 
Hoards 



Percentage of 
All 119 
Respondirg 
Hoards 



Percentage of 

38 Boards 
with Director 
Policies 



Preparat'.on 

Purpost 3 

development/ 
improvement : 
tfpneral 12 

development/ 
improvement : 
specific 6 

administrative : 
general 3 

administrative: 
specific 7 

Criterion domains 

administrative 
performance 9 

school and 
community 

relations 10 



10.1 

5.0 
2.5 
5.9 

7.6 
8.4 



31 .6 

15.8 
7.9 
18.4 

23.7 
26.3 
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Table 28 (continued) 



Component 



Number of 
Boards 



program 

organization 6 
personnel 

management 6 

contribution 

to board 7 

Sta'.iards 0 

Job description 7 

Activities prior 
to appraisal 

request for 
statement about 
objectives, 
acti^ *ties and 
plans 8 

request for 
self-evaluation 2 

pre-conference 13 



interim 
conference 



13 



Objective-setting 30 

Data Collection 

Data collection 
methods 

examination 

of documents 1 

interview 10 

Sources of 
information 

director 
being 

appraised 18 
trustees 2 



Percentage of 
All 119 
Responding 
Beards 



5.0 
5.0 

5.9 
0.0 
5.9 



6.7 

1.7 
10.9 

10.9 
25.2 



0.8 
6.4 



15.1 
1.7 



Percentage of 
38 Boards 
with Director 
Policies 



15.8 

15.8 

18.4 
0.0 
18.4 



21 .1 

5.3 

34.2 

34.2 
78.9 



2.6 
26.3 



47.4 
5.3 



ERLC 



93 

lio 



Table 28 (continued) 



Component 



Number of 
Boards 



Percentage of 
hU 119 
Responding 
Boards 



Percentage of 
38 Boards 
with Director 
Policies 



Appraiser(s) 

direccor 
being 

appraised 2 

all trustees 12 

committee of 
trustees 17 

Tine spent 

trequency 24 

Reporting and FoIIow-up 

^O3t-conf erence 1 7 

Written report 12 

Destination of 
report 

director 4 

trustees 14 

Follow-up 

plan developed 4 

monitoring 

of plan 3 

Differentiation of 
Procedures 0 

Evolution of Policy 

Development of 

policy 2 

Implementation 

of policy 1 

Review of policy 2 



1 .7 
10.1 

14.3 

20.2 

14.3 
10.1 

3.4 

1 1 .8 

3-4 
2.5 
0.0 



1.7 
0.6 

'.1 



5.3 
31 .6 

44.7 

63.2 

44.7 
31.6 

10.5 
36.8 

10.5 
7.9 
0.0 



5.3 
2.6 

5.3 
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Preparation 



Purposes 



The discussion of the purposes of director appraisal was, as for superintendents, very brief. 

T^^ director level is the only one in which a greater percentage of boards listed 
administrative rather than developmental/improvement reasons as the specific purposes for 
appraisal . 

Of the seven boards indicating a specific administrative purpose, five specified that 
appraisal was used in salary decisions. 

Criteria, Standards, and Job Descriptions 

In general, policies for director appraisal are very similar to those for superintendent 
appraisal, and this is definitely so with respect to criteria and standards. There is again 
no evidence of rigorously defined standards and, in comparison with the situation for teachers 
and principals, not much attention paid to specifying criteria. Figure 16, included later as 
part of the Data Collection section, illustrates one of the more complex sets of criteria for 
directors. 

The tendency, noted in superintendent policies, to deal with criteria by employing th*^ 
notion of management skills rather than by outlining lists of expected behaviours in 
functional criterion domains was even more marked among director policies, as was the tendency 
to list personal qualities as indicators of an effective director. 

However, for directors, there was less use of the approach taken among superintendent 
policies that involved outlining duties and responsibilities primarily with respect to the 
various gr-iups with whom the superintendent worked. 

Among director policies, there was less frequent reference to job descriptions, and job 
descriptions themselves were less elaborate, than in superintendent policies. 
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Although, as tables 27 and 28 indicate, there are some similarities between pre-appraisal 
practices in director and superintendent policies, the pattern of components has some 
differences. Half the boards submitting superintendent information specified that a 
pre-conference be held and only 6.5 per cent soecified that the superintendent prepare a 
statement about objectives, activities, and plans. In contrast, only 34.2 per cent of the 

^ards submitting director information required a pre-conference to be held, but 21.1 per cent 
required a prepared statement. It appears that, in comparison with the procedures for 
superintendents, there is some tendency for directors to be required to prepare a statement of 
objectives on their own, prior to any meeting with their appraiser. The collaboration 
regarding objective-setting between appraiser and appraisee, which is the usual business of 
the pre-conft ance for all other staff categories, takes place somewhat less frequently for 
directors. When it does occur, it is in some cases done only after the director has made a 
formal presentation of objectives to the boat 

Objective-Setting 

As is the case in superintendent policies, objective-setting is a very commonly specified 
component of director policies. Although some features (specifically, the cyclical nature of 
the r ocess and the fact that interim modification of objectives is possible) of director 
objective-setting are similar to those for superintendents, there are two ways in which 
objective-setting for directors is unique. 

First, there is somewhat less indication that objective-setting is to De a collaborative 
process, with only about one-third of the 30 boards that report objective-setting also 
specifying such collaboration. As mentioned earlier, in many cases the director is solely 
responsible for setting goals which are presented to the board. In some such situations, the 
quality of goals set, as well as the extent to which they are later fulfilled, are considered 
in appraising the director. 

Second, although the policies of a certain number of boards specify that the objectives 
SPt by the director must be consonant with those set system-wide by the board, in other boards 
the objectives that the director sets are apparently used as overall system goals. 
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Data Collection 



Methods 



As is seer, in table 28, of the 11 boards specifying the method used to collect data for 
directors, all but one included an interview technique. A smaller percentage of boards, 
however, specified this component for directors than did for other categories of staff. 

Sources of Information 

Only two boards indicated that sources other than the director were consulted by those 
appraising the di'-ector. In both of ohese boards, all trustees were asked to complete an 
evaluation form and to retu'^n it to the chairman of the board. In one case, the chairman 
collated the responses and prepared a summary for use at the evaluation meeting; in the other, 
all completed evaluation forms were given to the director, who prepared a summary. Figure 16 
includes the form used by one of the boards. 

Figure 16: Trustee Evaluation Form for Director Appraisal (Policy Excerpt) 



1. EFFE C TIVE COMMUNICATION 

1. 1 Communication with Trustees 
The Director has: 

1. made trustees aware of the on-going progress of 
projects or areas of concern; 

2. made trustees aware of on-going implementation of 
policy progress and problems, prognosis; 

3. made trustees aware of current trends, legislation, 
forces, provincial policies; 

4. made trustees aware of al 1 a 1 ter natives and 
implications of decisions at Board level; provided us 
with sufficient information for making decisions; 

5. been accessible to hear individual trustees' concerns; 

6. made trustees aware of personal concerns and 
convictions, as well as those of senior administration 
and system. 

1. 2 Communication with the Community 

1. maintained contact with the community; 

2. effectively responded to concerns and issues in the 
public forum; 

3. been personally accessible. 



DIRECTOR PERFORMANCE REVIEW 



AREAS OF ASSESSMENT 



High 



Med. 



Low 
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Figure 16 (continued) 



1. 3 Communication with the System 

1. ensured the development and growth of a model of 
communication which allows for the flow of information 
and feedback; 

2. maintained contact with the system; 

3. provided for increase in awareness of all levels of 
the system; 

4. been personally accessible. 

2. EFFECTIVENESS IN THE MANAGEMENT OF HUMAN RESOURCES 
The Director has: 

2.1 provided clear roles and expectations for members of the 
administration; 

2.2 ensured that individuals are in roles suited to their 
strengths, interests, and abilities; 

2. 3 del egated authori ty rel ated to def i ned rol es and 
responsioi 1 ities ; 

2.4 carried out evaluation of senior administrative staff; 

2.5 provided opportunities and encouragement for individual 
growth and professional development; 

2.6 been conscious of enhancing satisfaction and motivation of 
staff; 

2.7 involved staff appropriately in decision-making processes 
and system planning and development; 

2 8 facilitated co-operative effort between individuals and 
within and across different groups in the system; 

2.9 been sensitive to the thoughts and feelings of others, and 
understanding of the different perspectives of other 
individuals and groups; 

2.10 successfully managed difficult personnel issues; 

2.11 provided a supportive climate for people; 

2.12 demonstrated both fairness and firmness; 

2.13 ensured that effective processes for the management of 
human resources are carried out throughout the system; 

2.14 been conscious of providing for his own job satisfaction 
and professional development. 

3. EFFECTIVENESS IN TOLICY DEVELOPMENT 
The Director has: 

3.1 identified needed areas for policy development; 

3.2 anticipated future needs; 

3.3 set priorities for policy development; 

3.4 ensured thorough development of policy; 

3.5 ensured effective policy implementation; 

3.6 ensured effective evaluation of new and continuing 
pol icies. 

4. ORGANIZATIONAL SKILLS; TECHNICAL KNOWLEDGE 
The Director has: 

4.1 developed an organizational structure which functions 
effectively in meeting the needs of the system; 

4.2 demonstrated necessary knowledge of every aspect of the 
educational system, and awareness of issues affecting 
teaching and learning; 

4.3 ensured effective management and co-ordinacion of each 
facet of the educational system; 

4.4 demonstrated the ability to implement ideas and effectively 
institute change; 

4.5 provided for the development and implementation of 
long-range plans; 

4.6 provided for a sense of direction in the system. 
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Figure 16 (continued) 



High Med. Low 

5. CONCFPTUAL SKILLS 

5.1 is able to look at issues from many diverse perspectives 
and, therefore, is able to move beyond conventional forms 
of problem solving and into an arena which allows for a 
great deal of creative thinking in working through 
situations; 

5.2 is constantly thinking and searching for new and creative 
ideas to enhance the life and vitality of the school 
systen; 

5.3 exhibits a keen understanding of self, others, and the 
broader society; 

5.4 reads a wide variety of materials and has a comprehensive 
view regarding ideas and philosophies of education and 

1 i f e ; 

5.5 is excited about the world of ideas and is constantly 
searching for higher levels of personal insight and 
meaning. 

6. PERSONAL QUALITIES 
The Director: 

6.1 performs well under pressure; 

6.2 copes successfully with the continuing stress and demands 
of the position; 

6.3 demonstrates personal strength and confidence; 

6.4 is decisive, yet can be flexible when warranted; 

6.5 demonstrates good judgement; 

6.6 maintains personal integrity; 

6.7 demonstrates high commitment to education and to the needs 
of students; 

6.8 is able to deal realistically with problems yet keeps a 
positive perspective. 



DATE TRUSTEE'S SIGNATURE 



Appraiser(s) 



With the exception of two boards with policies specifying director self-evaluation as an 
initial stage in the appraisal process, there were only two types of appraisers indicated for 
directors. Twelve boards called for the entire board to evaluate the director. In 17 other 
boards, a small committee of trustees, usually called the Director's Review Committee, was 
to appraise the director and then report its findings to the entire board. 
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Time Spent 



As was the case for teachers, principals, and superintendents, one of the most often specified 
components of director appraisal was the frequency with which it should be done. 

Reporting and Fol low-Up 

In director appraisal policies the pattern of components regarding post-conferences, reports, 
their destinations, and follov^-up was very similar to that for superintendents. The sole 
exception to this was in the area of written reports on appraisals. Of the 12 boards 
specifying d written report, nine clearly indicated the form the report was to take. Three 
boards specified an anecdotal report, and six some sort of checklist or rating scale to 
appraise directors' personality traits, behaviours, or, in one case, objectives. 

Differentiation of Procedures 

None of the boards indicated that their appraisal procedures were differentiated in any 
manner. 

Evolution of Policy 

The number of boards providing information about the development, implementation, and/or 
review of their director appraisal policy was somewhat less than those providing analogous 
information for superintendents, and considerably less than those doing so for teachers and 
principals. 



Summary 



Results of the screening questionnaire and the policy analyses, taken together, indicate that 

appraisal policies are most widespread and most detailed for the teacher role. They become 

progressively less widespread and less detailed for principal, superintendent, and director 
roles. 
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Whe;i the components of the policies submitted for the four roles are examined, a few 
differences emerge clearly. 



Prepa.'ation 

With respect to preparation for appraisal, teacher policies included the most detail regarding 
the purposes of appraisal. Although policies for all roles except the director indicated that 
developmental/improvement purposes were more important than administrative purposes of 
appraisal, the emphasis was particularly marked for teacher policies. 

The criteria by which appraisees were judged were much more frequently specified in 
teacher and principal policies than in those for superintendents and directors. For the 
latter roles, particularly for superintendents, there was more use of job descriptions as a 
mode of specifying expected performance. 

While pre-conferencing was specified to some exient in all policies, there was also the 
requirement in some principal, superintendent, and director policies that the appraises submit 
some sort of statement or documentation to begin the appraisal process. This requirement was 
most common for principals, where a substantial number of boards expected specific sets of 
information about the school to be submitted to the apprai ser(s). 

The setting of objectives on which performance could be appraised occurred at all levels, 
but was much less frequently included in teacher policies than in those of administrators. In 
addition, some of the objective-setting specified in teacher policies focussed on lessons, 
rather than on long-term plans. 

Data Collection 

Judging from the policy information submitted, we see that methods of collecting data for the 
appraisal of teachers and administrators differ greatly. Teachers are evaluated primarily 
through observation while the others are appraised primarily through interview techniques. 



1 see. 



Most appraisal involves collecting information almost entirely from the apprai: 
However, a number of principal policies specified that information about appraisees should be 
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collected from a variety of other sources: teachers, department heads, vice-principals, 
superintendents, parents, and students. 



The identity of the appraiser is perhaps the most widely specified of all components of 
appraisal systems and varies, naturally, with th'; role involved. In most situations, 
appraisal is conducted by individuals, but there are two exceptions. A substantial number of 
boards specified that an evaluation team be used for principals, and directors are appraised 
either by a small committee of trustees or by the entire group. 

The frequency with which appraisal takes place is a commonly specified component of all 
policies. Some teacher and principal policies also included an indication of '.he duration of 
the appraisal process. 

Reporting and Fol low-Up 

A considerable proportion of the boards submitting appraisal information specified a 
post-appraisal conference and the form the appraisal report should take. There also was 
relatively frequent specification of the destination of the appraisal report for teachers; 
this was less frequent for other roles. 

The percentage of boards mentioning that a post-appraisal plan be adopted and monitored 
declines progressively from teacher, through principal and superintendent, to director. 



Differentiation of Procedures 

Teacher policies differ from those for principals, superintendents, and directors most 
markedly in the differentiation of procedures, with teacher policies including substantially 
more differentiation regarding appraisal procedures. There is differentiation between 
permanent and probationary staff, satisfactory and unsatisfactory teachers, and procedures 
used for different purposes and/or facets of evaluation. 
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Evolution of Policy 



Again, information on teacher appraisal policies includes substantially more detail about the 
development, implementation, and review of such policies than is the case for the other roles. 

From the number of submitted policies and their complexity, as well as from the screening 
questionnaire data, it is clear that more attention has been paid across Ontario to the 
formulation of policies for teachers than for administrators. Among administrators, it is 
apparent that principal appraisal has received the most attention, and director appraisal the 
least. 
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CHAPTER 3 

SURVEY OF PERFORMANCE APPRAISAL PRACTICES: DESCRIPTION OF FINDINGS 



A major survey undertaken as part of this study of performance appraisal policies and 
practices in Ontario school boards had four major objectives: (1) to describe current 
practices in detail as they were perceived by teachers, principals, superintendents, 
directors, and trustees; (2) to desc.ibe the opinions of these groups concerning appraisal 
practices they believed ought to be followed; (3) to determine what performance appraisal 
practices are most effective; and (4) to provide a data-base to answer addtional questions 
concerning performance appraisal practices. 

This Lhapter describes the results of the survey and is divided into eight major 
sections. The first describes in considerable detail the methods used to conduct the survey 
and the nature of the resulting data-bases. The next four describe, in turn, the results of 
the survey as they relate to teachers, principals, superintendents, and directors. The sixth 
section provides a demographic profile of the respondents, the seventh data on dismissals and 
grievances, and the last conclusions and implications. Data from the survey of trustees are 
related in sections concerning teachers, principals, superintendents, and directors. 
Information on the evaluation of those in other roles within the educational system were not 
collected in the survey. 

The data reported in this chapter were selected, for the most part, from Appendix A in 
Voluipe 2 of the Technical Report of this project. This appendix consists of sets of tables 
describing the results for each question in the survey, broken down by type of school (public 
elementary, public secondary, and separate). In most cases, only the overall results for the 
English-language survey are presented in this chapter, though important differences between 
respondents in different types of school are noted, as are differences between anglophone and 
francophone respondents. 

In presenting the results in the four sections concerned with performance appraisal 
practices, we use the same framework as in the other major chapters of this report. Before 
proceeding to the findings, however, we describe the survey design. 
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Survey Methods 



Carrying out the survey involved four tasks: selection of the sample, development of the 
questionnaire, administration of the questionnaire, and analysis of data. 

Design of Sample 

The sample of school boards selected for the the questionnaire survey was chosen to satisfy a 
number of criteria. First, the sample had to represent school boards (1) from all regions of 
Ontario, (2) of both types (public and separate), (3) of all sizes, and (4) with different 
types of performance appraisal systems. Second, it had to be of sufficient size to allow 
reasonably reliable inferences using parametric statistics. Third, it had to be adequate for 
one to make equally reliable statements about public aid separate boards. Finally, a 
sufficient number of boards with significant numbers of francophones had to be selected to 
provide some information about performance appraisal systems affecting francophone educators. 

Samples of individuals within boards also had to meet several criteria. First, tneir 
numbers could not total more than 6000 because of cost considerations. Second, a sufficient 
number had to be selected from each board for reliable estimates to be made of school board 
characteristics that required aggregate data. And third, it was desirable to be able to 
measure variation among schools within boards with reasonable reliability. 

Data collection to select our sample has been described in detail at the beginning of 
chapter 2. For the reasons noted, we decided to concentrate only on 77 public and 49 separate 
school boards. 

Three boards did not respond and were excluded from the study. Five boards requested 
exemption from later participation in the study on various grounds; prominent among these was 
that they were currently introducing a new performance appraisal system and felt participation 
in the study would not be appropriate. As well, we excluded the board selected for pilot 
testing all instruments and the case study methodology. Thus, we had a final population of 
117 school boards, from which the sample was drawn. 
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To achieve the objective of maKing reliable inferences about all school boards, we 
selected a target sample size of 30, this being the minimal size for which one normally use? 
parametric statistical methods. With the expectation that about 15 per cent of the boards 
selected would decline to participate, we planned a sample of size 34. 

All the variables noted in our first set of criteria above were incorporated in the 
design of the sample. In particular, the population was stratified by type (74 public and 43 
separate boards) and region (43 northern, 18 western, ^1 central, and 15 eastern boards). In 
the case of region. Ministry of Education definitions were used, with the northwestern, 
midnorthern, and northeastern being combined into one northern category. In all, there were 
eight cells from which subsample? were drawn. 

To determine the number of boards to be drawn from each cell, sampling with probability 
proportional to size (PPS) was planned. This technique has a number of advantages. In this 
case, the most prominent was that the number of boards selected from each cell would reflect 
not the total numher of boards in each cell but the relative size of the educational 
enterprise of all boards in each cell. Thus, the large number of small northern boards would 
not swamp the sample. 

To ensure approximately equal numbers of public and separate boards, which was necessary 
if statements of equal reliability were to be made about them, we had to weight the sample 
selected so that puDl z boards would be "undersampled" and separate boards "oversampled". In 
particular, a weig t of two-thirds was used in selecting public boards (i.e., two-thirds the 
number were selected than otherwise would have been selected) and a weight of two was used to 
select separate boards. 

The basic elements of the sampling design are displayed in table 29. In it, sampling 
with simple PPS is displayed in column 4 and PPS with weights in columns 5 (public boards) and 
6 (separate board^/. Note that enrolment was selected as the measure of board size. 
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Table 29: Sample Design for Performance Appraisal Survey 



Boarci 




1 


2 


3 


.a 
4 


5 


6 


/ 


Type 


Region 


No. 


Enrol. 


Enrol. 




#4 


#4 










000* s 


% 


x35 


x2/3 


x2 




Public 


North 


27 


131 


7.9 


3 


2 


— 


4 








1 7Q 


1 U« / 


4 






2 




Central 


28 


817 


48.9 


1 7 


1 1 




8 




Eastern 


9 


150 


9.0 


3 


2 




2 












27 


18 




16 


Separ* 


North 


17 


62 


"5.7 


1 








ate 


Vedtern 


7 


49 


2.9 


1 




2 


2 




Central 


1 r 


238 


14.3 


5 




10 


7 




Eastern 


6 


43 


2.6 


1 




2 


2 






117 


1670 


100.0 


8 




16 


14 



Coluinn 4: As proposed in contract. 
Coluom 5^6: As revised. 
Column 7: Actual sample. 



To select the boards within each cell, either (1) PPS with implicit stratification and 
systematic sampling or (2) deep stratification with PPS was used. In the first instance, 
appraisal system (implemented system versus non-implemented system) and size were used as the 
variables to order the boards. In practice, all boards within each cell we'^e first placed 
i nto two groups based on the responses in the screening questionnai res regard i ng the 
implementation and effectiveness of teacher appraisal systems recorded on the screening 




questionnaires. If scores, which ranged from 0 to 6. were 4 or below and a review of the 
policy showed it did not have clearly defined objectives, then the board was placed in the 
non- implemented category. If the score was greater than 4 or was 4 with clear objectives, it 
was placed in the implemented group. Within each of these groups, boards were ranked by size. 
Then a PPS sample was drawn using an appropriate interval to produce a sample of v.tc requisite 
number of boards from that cell. Tables of random numbers were always used to select the 
first board in the subsample. 

Deep stratification was used when there were two boards to be selected since the other 
technique could easily have produced a sample of two large schools. With deep stratification, 
pairs of boards were formed that were at the extremes, e.g., a large board without an 
appraisal system and a small board with one. Then, probabilities proportionate to the 
combined size of each pair were computed and a random selection of a pair was made. 

To ensure an adequate sample size, we decided that, if the number of boards agreeing to 
participate fell below 28, backup boards would be used. For this purpose, the ordered lists 
prepared for the sampling process were used, and the board on these lists that preceded a 
board which declined to participate was selected. In practice, it was necessary to use these 
boards as the rate of refusal on the first round was about 30 per cent rather than the 15 per 
cent projected. The breakdown of the final sample of boards by type and region is displayed 
in column 7 of table 29. 

Having selected the boards, we next selected individuals within boards to be sent 
que Aionnaires. These were in five categories: teachers, principals, superintendents, 
directors, and trustees. Since we used different procedures to select subjects in the 
English- and the French-language surveys, it is necessary to outline the procedures used in 
each. 

To select teachers in the English-language survey, cluster sampling using schools as 
clusters was called for to make data collection easy and to make it possible tc estimate 
v;^riation among schools within boards. The total number of teachers to be selected numbered 
about 5000; this number was considered sufficient to produce approximately 3000 returns 
(allowing for a 60 per cent rate of return). If these were spread uniformly among all boards, 
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there would be 100 respondents per board on the average. In the case of boards of education, 
this would provide about 50 respondents on average for each of the elementary and secondary 
panels. 

We deemed inadequate the uniform allocation of questionnaires to boards suggested by the 
average figures, since it would provide little reliable information auout differences among 
secondary schools in larger boards and more information than might be needed about variation 
among individual schools within boards. Indeed, in a number of boards in the sample, 
virtually all teachers would have been included. An alternative was to conduct a PPS sample 
of teachers. However, we discarded this option since only a handful of teachers would be 
included in the smallest boards and about 500 teachers in the largest board in the sample. In 
the latter case, this would have required sending surveys to all teachers in about 30 schools, 
a number which would have imposed too great a burden on one school board. 

In practice, we compromised between the two extremes of PPS and uniform allocation. In 
the case of separate school boards and the elementary panels of boards of eo^-cation, we 
selected 20 per cent of the schools if this number was between four and 16 inclusive; if it 
was smaller, we selected four schools (or all, if there were fewer than four); if it was 
larger, we selected 16. For secondary schools in boards of education, we selected 50 per cent 
of all schools if this number was between two and eight; if it was less than two, we selected 
two schools (or one, if there was only one); if it exceeded eight, we selected eight. Wt 
omitted private secondary schools associated with separate schools, as well as all 
French- language schools, the latter being surveyed separately. 

The selection of schools within boards was done using systematic sampling with the 
Ontario Ministry of Education's Directory of Education, 1982/83 used as the sampling frame. 
In each case, a sampling interval k was selected by dividing the number of schools to be 
selected into the number of English-language schools in the board. Then, after a random 
start, every kth school was selected. 

In the case of elementary and separate schools, questionnaire were sent to all teachers 
in the school. Questionnaires were also sent to all teachers in secondd.^ ^chools with fewer 
than 40, teachers. In secondary schools with over 40 teachers, principals were asked to use a 
staff list to select every second teacher. (There were two exceptions to this practice, and 



both occurred in a single board. In one school with a staff of 75, at th^ principalis request 
all teachers received questionnaires; in another school with a staff of 155, at the 
prlncipaTs request only one-quarter of the teacher^ received questionnaires.) Assuming a 60 
per cent rate of return, we expected a minimum of 16 questionnaires per school. 

Principal questionnaires were sent to all principals in all but the two largest boards in 
the sample. For these two boards, questionnaires were sent to all principals of schools in 
which teachers received questionnaires and to a fraction of the remaining principals. In ont 
of the two boards, principals of every third remaining (i.e., not involved in teacher sample) 
school received questionnaires; in the other board, principals of every sixth remaining school 
received questionnaires. 

All academi: superintendents were selected in school boards with 20 or fewer 
superintendents and approximately half of the superintendents were selected in larger boards. 
(The selection of only a portion of the superintendents took place in only two boards, the 
same two that did not include in the sample all principals In the board.) The primary concern 
in the case of superintendents was not to overburden a board's administrative staff rather 
than to achieve some minimal number of returns. 

All directors in the 30 boards selected were included in the sample. Each of these was 
asked to selert five trustees who were knowledgeable about performance appraisal procedures in 
their board and to forward questionnaires to them. Given that new school boards had taken 
office only a few months before the survey, we deemed random sampling methods unsuitable. 

The final composition of the sample included 5655 t-.achers, 1211 principals, 214 
superinLendents, 30 directors, and 150 trustees. 

Included within the English-language survey were the directors and superintendents in the 
two boards that had agreed to participate in the French-language survey. Hence, 
French- language surveys were sent only to teachers, principals, and trustees. In particular, 
all teachers and principals in French-language schools in these boards received 
questionnaires; as well, the directors were sent five French-language trustee questionnaires 
to distribute to francophone trustees they believed to be interested or knowledgeable about 
the performance appraisal of board staff. 
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Development of Questior naire 



Developing the survey instruments was a laborious but straightforward undertaking that 
occurred in four stages: (1) review of existing policies to develop a framework for the 
questionnaires; (2) writing of items to assess the presence or absence of practices identified 
in the review of policies; O) pre-test of the questionnaires in a pilot study; and (4) 
revision of questionnaires to resolve problems found in the pre-test. 

The framework for the questionnaires was outlined in the preceding chapter. In addition 
to questions relating specifically to practices, items were added that were concerned with 
policy development, implementation, and review, and that measured various demographic 
variables thought to be related to the performance appraisal of staff (e.g., experience in a 
given position). 

The questionnaires were designed to measure both the general perceptions of respondents 
{e.g., how helpful a given practice is in general) and specific perceptions (e.g., how helpful 
a specific practice was in an individual's last performance appraisal). This distinction 
required the use of numerous screening questions to ascertain which practices a given 
evaluatee (or evaluator) had experienced. 

Pre-test results were generally positive, though the length of the questionnaires was a 
concern. Multiple-choice items were therefore redesigned as checklists, and the number of 
auestions each role incumbent was asked about the evaluation of those in other roles (e.g., 
teachers' opinions and experiences concerning principals) were reduced. The final 
questionnaires are reproduced in Appendix B of Volume 2 of this Technical Report. 

For the French-language survey, the final versions of the English-language instruments 
were translated by a native speaker of Fren:h who was fluent in English. These questionnaires 
were vetted by a f ranco-Ontarian supervisory officer. Only a few changes were suggested. 
These were made, and the questionnaires printed. 
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Questionnaire Administration 



Directors of the school boards selected were telephoned to request their participation in the 
questionnaire phase of the study; these calls were followed by letters requesting that a 
liaison person be selected with whom the researchers could co-ordinate delivery and collection 
of the questionnaires. 

In a typical board, all questionnaires were sent by courier to the contact person, who 
then distributed them to the schools and personnel indicated in a letter of transmittal. A 
brief abstract of the study's purposes was provided so that each administrator would be 
adequately informed about the project. 

The liaison person also collected the completed questionnaires, which were sealed in 
envelopes to protect confidentiality, and returned them by courier to the project staff at 
OISE. 

Departure from this standard procedure occurred in several cases. In two boards in or 
near Toronto, teacher and principal questionnaires were sent directly to schools so as to 
reduce the administrative burden to these boards. The completed questionnaires were also 
returned directly to OISE. 

Overall, this process worked well and data collection for the English-language survey was 
completed between May 1 and June 30, 1983. 

For the French-language survey, the process of questionnaire administration was similar. 
However, only two of the three boards requested to participate agreed to do so. 
Questionnaires were sent to these two boards in mid-October 1983, and completed questionnaires 
were returned by mid-Decembe**. 

Analysis of Data 

Data analysis occurred in three phases: data preparation, descriptive analysis, and 
correlational analysis. On receipt, questionnaires were vetted by project staff tt ensure 
codable responses were given. Data were then keyed, either onto tape by a commercial 
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key-punching firm or by project staff using OISE computer facilities. After a complete data 
set was available for a given group, data were run through editing programs to ensure the 
number of records for each subject was correct, and that al^ responses fell within acceptable 
ranges. When this was not the case, corrections were made or cases were omitted, depending on 
the magnitude of the error. 



Descriptive analysis of the data sets, presented in Appendix A of Volume 2 of this 
Technical Report, were prepared. These tables display the distributions of responses to each 
questionnaire item broken down by school type (public elementary, public secondary, or 
separate). Separate sections are devoted to the results for each group surveyed for both the 
English- and French-language surveys. Thus, there are eight sections in all: for the 
English-language survey, teachers, principals, superintendents, directors, and trustees- for 
the French-language survey, teachers, principals, and trustees. For the French-language 
responses, it was not deemed advisable to display data by type of school since there »'ere but 
two boards, one separate and one public. To have done so would have allowed identification of 
an individual board's response, in violation of the agreement to protect confidentiality. 

Two technical issues arise in the presentation of survey results for a survey with a 
complex sampling design such as is the case in this study: What is the proper unit of 
analysis? And, how should the responses be weighted? 

In this study, at least three possible units of analysis could be used, the individual, 
the school, or the board. The results are not necessarily tht same, though they are often 
similar. Of the three, we chose t',o. The individual, as noted, was used as the unit of study 
in the tables reported in Appendix A, Volume 2. As well, to analyse the effect of specific 
board policies, the board was the appropriate mit of analysis since policies affect all 
individuals in one board, but not those in other boards. 



The issue of weighting the sample results is relevant regardless of the unit of analysis. 
Two primary choices exist: (1) no weighting (or, equivalently, weighting each response 1) or 
(2) weighting inversely to the sampling ratio (i.e., 3/2 or 1.5 for public boards and 1/2 or 
0.5 for separate boards, since public boards were undersampled by 1/3 and separate boards were 
oversampled by a factor of 2); as well, at the individual level, one may (3) weight by size 
(e.g., enrolment) or (4) by a combination of size and sampling ratio. 
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To provide an idea of the effects of each choice, we estimated the percentage of teachers 
with a degree, using first the board and then the individual as the unit of analysis, the 
first case, one is estimating the average of the percentage of teachers in Ontario school 
boards with degrees; in the second, one is estimating the percentage of teachers in Ontario 
with degrees. The estimates are given in table 30. 

Table 30: Effect of Weighting on Estimates of the Percentage of Teachers w ith Degrees, Using 
Board and Individual as Units of Analysis 

Unit of Weighting Scheme 

Analysis Unweighted Weighted by Weighted by Weighted by 

Enrolment Inverse cf Enrolment and 

Sampling Inverse of 

Ratio Sampling 
Ratio 

Board 69. 8? N/A 71.6$ N/A 

(9.04)* (7.72) 

Individual 72.8^ 72. 9$ 74.2 74.7? 

(7.63) (7.24) (6.45) (6.15) 

* Standard deviation in parenthesis. 

^ Equivalent to weighting board average by the number of respondents. 

Note that at the board level the unweighted estimate is 1.8 percentage points below that 
of the weighted estimate using the inverse of the sampling ratio. The bias of the unweighted 
estimate results from the proportionately larger number of separate school boards in the 
sample; separate school teachers are less likely to have degrees than teachers in public 
boards of education, because of the very high percentages of degree holders among secondary 
school teachers in the boards of education. In the sample, 83.7 per cent of all sec ndary 
teachers held degrees, while only 65.8 per cent of the public elementary and 67.6 per cent of 
the separate school teachers did so (see table 1.88 in Appendix A, Volume 2). 

The four estimates for teachers with degrees are all higher than the average percentages 
of teachers with degrees in school boards because larger, urban boards tend to have higher 
percentages of teachers with degrees than do many smaller boards; hence, at the individual 
level, the percentage with degrees is larger. Weighting by the actual enrolment of these 
boards (rather than, in effect, depending on the sample size to provide the weight) makes 
little difference, even though the sampling strategy tends to undersample teachers in large 
boards and to oversample those in ^mall boards to ensure equally reliable estimates of school 
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board parameters. However, using the inverse of the sampling ratio does raise the estimate 
1.4 percentage points above the unweighted estimate; using both that and enrolment raises it 
a further 0.5 percentage points. According to statistical theory, this latter is the best 
unbiased estimate of the percentage of Ontario teachers with degre.-^. 

For the analysis in this report, we decided to use unweighted estimates, even though 
these are biased towards the characteristics of separate school boards and small boards. This 
decision was made for several reasons. First, we believe it easier for most readers to 
understand an average or percentage of a sample of respondents than a weighted average. 
Second, it was computationally less expensive, an important consideration given the large 
sample size and number of items. Finally, and perhaps most important, we felt that the 
unweighted estimates provided a better picture of the typical teacher or board in that a 
ceiling was placed on the sample sizes from large boards, and, hence, respondents from a few 
large boards did not swamp the sample; and that public and separate boards were about equally 
represented in the sample, and, hence, public board characteristics did not dominate the 
results. 

Unbiased estimates can, of course, be calculated from the data base should these be 
required. As well, for a reader interested in assessing the extent of bias caused by the 
oversampling of separate schools, a good idea can be had from the tabies in Appendix A, Volume 
2, wherein data are presented separately by school type. 

In addition to the presentation of responses by school type, we made board level 
investigations of the characteristics of effective evaluation systems for teachers and 
principals. Again, we used unweighted means and percentages for reasons similar to those just 
outlined. This analysis used Pearson-product moment correlation coefficients. The 
methodology is explained in further detail in the relevant sections. 

The two types of analysis carried out are but a fraction of those possible. The data 
base could be queried on many questions related to performance appraisal. Though the 
requirement that responses remain confidential precludes release of the data base, a brief 
outline of its structure may be of use to those who might request specific analysis from the 
authors of this report. 
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Data are in two forms: raw data and SPSS-X SAVE FILES. The raw data for the eight 
questionnaires are stored on tape to which only the authors have access. These are viewed as 
backup files in case of a loss of the SPSS-X files. 

There are 10 SPSS-X SAVE FILES in all, stored in archive on the OISE VAX-750 statistical 
computer in account PERFORMSL. These files include complete documentation including variable 
names, variable labels, value labels, and field locations for all questionnai'^e items. For 
the major survey, item names are keyed to questionnaire numbers to facilitate analysis (e.g., 
TEACH5D is item 5D on the teacher questionnaire). The save files are named as follows, their 
content is self-explanatory: TEACHER. SSF; PRIN.SSF; SUPT.SSF; DRCT.SSF; TRS.SSF; TEFR.SSF; 
PRFR.SSF; TRSFR.SSF. For the screening questionnaire, the responses for all boards are in 
file PASNEW.SSF, and for the sample of 30 boards in the study, PASSAM.SSF. 

Results 

Performance Appraisal of Teachers 

A sample of 5655 teachers in 30 Ontario school boards 16 public and 14 Roman Catholic 
separate — was asked to reply to the English-language school (ELS) questionnaire on teacher 
evaluation. From two scliool boards, one public and one separate, an additional 201 teachers 
in schools where French is the language of instruction (FLS) were asked to complete the 
French-language version of this questionnaire. 

A total of 4082 teachers completed the ELS questionnaire, for a return rate of 72.2 per 
cent, and 139 teachers completed FLS questionnaires, for a return rate of 69.2 per cent. All 
the French-language respondents were from Eastern Ontario. The distribution of ELS 
respondents by region and type of school is presented in table 31. The Total column at the 
right includes 14 respondents for whom type of shcool was not available, and excludes one for 
whom region was missing; explanations for other minor discrepancies in numbers are accounted 
for in Appendix A, Volume 2 and, for brevity's sake, are omitted here. The Total row at the 
bottom provides the percentage distribution by type of school. 
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Table 31: Distribution of ELS Teachers by Region and by Type of School 



Region 



Northern 
Western 
Central 
Eastern 

Total 



Public 

Elementary 

(n-1352) 

n % 

179 no 

189 14.0 
811 60.0 
173 12.8 

1352 33.1 



Public 

Secondary 

(n-1458) 

%_ 

149 10.2 

200 13.7 

936 64.2 

173 11.9 

1458 35.7 



Separate 

(n-1257) 
n % 
163 13.0 
209 16.6 
769 61.2 
116 9.2 

1257 30.8 



Total 
(n=4081 ) 

,_JL %_ 

491 12.1 

598 14.6 
2529 62.0 
463 11.3 

4081 100.0 



In the subsections that follow, the experiences and opinions of these four-thousand plus 
teachers are reported. All analyses except those concerned with explaining the effectiveness 
of different aspects of teacher evaluation practices use the individual teacher as the unit of 
analysis; the other analyses use the school board as the unit of analysis. Except when 
otherwise indicated, data in tables refer to the findings in English - language schools. 



Preparation 



Preparations for teacher evaluation may be as simple as a principal's informally dropping in 
unannounced while class is in session to "see how things are going", or as complex as a series 
of school-wide sessions to discuss objectives and agree on the criteria for evaluation. We 
discuss four aspects of preparation: planning, purposes, criteria, and standards. 



Planning 



The survey questionnaire included about a dozen items concerned with planning for teacher 
appraisal. Included vere questions about the rr^thod by which a teacher was notified of the 
impending formal evaluation, the existence and length of any pre-conferenc^, whether or not 
objectives were set, and the nature of objectives if they were set. Principals were also 
asked whether or not they set objectives with the last teacher whom they evaluated. 
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Table 32 presents the respondents' experiences as far as notification is concerned. 
Notification in person is most common, overall, but is most prevalent in public elementary 
schools. Memoranda are most likely to be used in secondary schools and are more likely to be 
used in separate than public elementary schools. Informal visits or observations in the 
classroom are likely to begin the process in almost half the cases in public and separate 
elementary schools. Respondents could check more than one option, so the percentages sum to 
more than 100. 



Table 32: Method of Beginning Teacher Evaluation by Type of School, as Reported by Teachers 



Method Public Public Separate Total 

Elementary Secondary 

(n»1105) (n=122l) (n=992) (n=3257) 

Yes Yes Yes Yes 



Notification 

in person 62. 5? 54-9? 51 .4? 56.5^ 

Notification by 

a memorandum 16. 3 47-7 25.1 30.5 

Informal visits 
or observations 

in classroom 48.5 20.6 49.0 38.2 



Thirty-six per cent of all teachers who had been evaluated in recent years indicated that 
a pre-evaluation conference took place with their evaluator. There was only slight variation 
among different types of school, with conferences being most common with secondary teachers 
(38 per cent) and least common with separate school teachers (32 per cent). These 
conferences lasted an average of 20 minutes, though they varied considerably, ranging from a 
minute or two to over an hour ( s. d. = 16 minutes). 

Francophone teachers' experiences were, for the most part, similar to those of their 
anglophone colleagues: 58 per cent were notified in person of their impending evaluation; 18 
per cent were notified by memorandum; but 67 per cent, higher than for anglophone teachers, 
experienced informal classroom visits. Thirty-eight per cent had pre-evaluation conferences 
and these were half-again as long as were those for anglophones, averaging 31 minutes ( s. d. = 
18 minutes. ) 
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As a result of pre-conferences, 71 per cent of the ELS teachers reported that they 
understood the criteria for evaluation; 75 per cent understood the purposes; and 52 per cent 
had agreed-upon objectives. Results varied little by type of school or language of teacher. 

A series of questions were asked regarding the setting of objectives in the teacher's 
last appraisal, independent of the question concerning pre-conferences. Overall, 23 per cent 
reported that objectives had been set. Percentages varied from a high of 28 for separate 
school teachers to a low of 19 for secondary school teachers; 22 per cent of francophone 
teachers reported this practice. 



Principals reported rather different behaviour in carrying out their last evaluation of a 
teacher: overall, 60 per cent of the anglophone and 42 per cent of the francophone principals 
reported having used objective-setting. 

Table 33 reports who set the objectives to be achieved -- the teacher being evaluated, 
the evaluator, or both collaboratively - as reported by teachers and principals. Teachers 
and principals concurred as to the ordering of these options in terms of frequency of use. 
Collaboration was most common, followed by the setting of objectives by teachers and then the 
evaluators. However, principals were far more likely to report having set objectives 
collaboratively than were teachers, and far less likely to report that the objectives had been 
set either by the teacher or by themselves alone. 



I^^^^ — Person(s) Setting Teach er Evaluation Objectives by Type o f School , as Reported by 
Teachers and Principals ^ 



Separate Total 

(n=256) (n=719) 
(n=16'?) (n=476) 



Person(s) 

Setting 

Objectives 


Role 

T 
P 


Public 

El era en ta ry 

(n«259) 

(n«250) 


Public 
Secondary 
(n=223) 
(n=64) 


TeLcher being 


m 
1 




28.6^ 


evaluated 


P 


10.0 


17.2 


Evaluator(s) 


T 


13.4 


16.1 




P 


S.4 


1 .6 


Collabora- 


Tl 


51.5 


44.8 


tively by 


D 


79.6 


79.7 


both 







32.0^ 

B.I 10.3 

19.1 16.4 

8.8 7.5 

46.5 .;7.6 

82.5 '30.7 
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The FLS situation was somewhat different. Onlv 40 per cent of the teachers reported that 
objectives were set collaboratively; as well, just 25 per cent indicated that they had set 
objectives themselves, and 35 per cent that their objectives had been set for them. FLS 
principals reported a different picture: 82 per cent indicated objectives had been set 
collaboratively and 18 per cent that the teacher had set his or her own objectives; none 
reported having set objectives for the teacher. 

Finally, it is worth noting that, among the three types of school, it is in secondary 
schools that teachers were most likely to set their own objectives. 

There was greater consensus between teachers and principals concerning the focus of the 
objectives. Seventy-eight per cent of the teachers and 81 per cent of the principals stated 
that the obj<*ctives focussed on overall teacher performance and not just on the lessons to be 
observed. Also, focus on overall performance was more frequently reported by separate school 
teachers (84 per cent) and principals (83 per cent) than their public school counterparts. 
The FLS percentages were yet more striking: over 90 per cent of both teachers and principals 
reported that objectives focussed on overall teacher performance. 

Purposes 

The evaluation of a teacher can serve many purposes, including the development of the teacher 
and the provision of information for administrative decisions. Teachers were* asked whether or 
not the purposes of their last performance appraisal had been clearly corimun^^cated to them, 
what the purposes of that appraisal had been, and what they believed the ideal purposes of 
teacher evaluation were. Principals were asked the last qu:stion as well. 

Overall, 71 per cent of all teachers, francophone and anglophone, indicated that the 
pjrposes of their last evaluation had been clearly communicated. About 23 per cent indicated 
they had not and the remaining 6 per cent could not recall. 

The purposes of these evaluations, and the purposes that teachers and principals believe 
teacher evaluation ought to serve, are reported in table 34. For the latter two sets of 
opini:ns, only those percentages for the "always should be used'' responses are reported. 
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l^^^^ 34: Actual and Ideal Purposes of Teacher Evaluation, as Reporte d by Teachers and 

Principals ^ ^ 



P^^P°3e8 Teachers 



Real Ideal 



Percent Percent 



To identify 
instructional 
weakness in need 

0^ improvement 46»7| 66,1| 



Principals 



Ideal 



(n«3257) (n-4082) (n-879) 



Percent 



Yes Always Always 



68. 3* 



To improve 
communication 
betweei: teacher 

and principal 28.5 45.1 40.1 

To comply with 
Ministry and Board 

Policy 76.3 40.4 37.7 

To qualify teacher 
for regular salary 

increases 3,7 12.6 2.6 

To identify teachers 
for possible 

promotion 7,8 30.2 15. C 

To identify inservice 

training needs 11.2 36.3 34.0 

To stimulate 
improvement in class- 
room performance 41.2 65.4 80.4 

To recommend 
probationary teachers 
for permanent 

contracts 19.3 69.7 57.2 

To identify teachers 
for layoff in case of 
redundancies due to 

enrolment decline 2.2 12.6 4.I 

To establish evidence 
for dismissal or 

retention 6.4 29. 1 21.6 

To assess effective- 
ness of instructional 

program 55.7 51 .2 73.6 

To identify 
individuals for 

transfer 2.2 8.6 4.8 

To reassure and 
develop teacher 

self-confidence 37.3 63.0 74.3 

To assess and 

improve curriculum 29.9 51.3 73.2 
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Several observations concerning table 34 are warranted. First, teachers appear to 
believe that the primary purpose of their most recent evaluation was to formally comply with 
policy, not to improve education in any way. Second, the most legitimate purposes of 
evaluation in the eyes of teachers are to decide on permanent contracts for probationary 
employees and to improve instruction. Third, principals are more likely to believe that 
teacher evaluation ought to play a role in the evaluation of programs and curriculum. 
Finally, teachers and principals agree that teacher evaluation ought not be used for 
administrative purposes such as selecting teachers for transfer or layoff, though teachers are 
more likely to take a hard line supporting such uses of evaluation than are principals. 

Responses to questions regarding purposes did not vary much among ELS public elementary, 
public secondary, and separate school teachers. However, FLS teachers were less likely to 
report that compliance with policies was the purpose of their last evaluation (about 46 per 
cent did so) and they were far more likely to believe that evaluation should be used for 
administrative purposes. For example, 28 per cent believed it should always be used to 
qualify teachers for regular salary increments, 32 per cent to identify teachers for 
promotion, 25 per cent to identify teachers for layoff, and 52 per cent to establish 
evidence for dismissal or retention. These attitudes were not as strongly reflected among FLS 
principals. Their figures for the preceding four items were 4 per cent, 12 per cent, 8 per 
cent, and 15 per cent respectively. 

Criteria 

Teachers were asked to indicate which among a set of 25 criteria were used in their last 
formal evaluation, the first six of which were general criterion domains with the remainder 
considered specific criteria. The teachers were also asked to state which of these criteria 
they believed always ought to be used in their evaluation. Their responses are reported in 
table 35. 
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^^^'^^ — Actual and Ideal Criteria Used in Teacher Evaluation, as Reported by Teachers 



Critprion -Pn-r 


Actual ly 


Ought Al 


Evaluation 


Used 


to Be Us 




(n-3257) 


(n=4082) 


x^a^ii^i. o L.J.CI001UUIU uxsTl OnnallC© 






±y ^m^iix, ±11 scnool anci comnuniuy 


57.3 


24.8 


Interpersonal relationqhi t^q 




37* 5 


Professions^ ipvplnnm<anf 


A Q A 


29*4 


Personal c^^uali . '^s 




46.4 




1 ^ 


9«6 


Punctuality 


35«0 


50 • 6 




1 8.9 


33.6 


'tf'f'co 1 Clinic ui ciassrooni 


66.2 


42.4 


n.\^ \^ Ui L a \^ jf UA XCUUXUS) 


36.4 


51 *2 


txsL ouiieij. appearance 


39^0 


'51^3 


xiivux vciucii ±n systeni— wiae activities 


31 -7 


12.1 


TpA (^}lP T*/ A rtm i ni a^rvo^-nr* n4--{ 

xcav^iici^ ciiiui J. iiA a u ra to r relations 


32.4 


17.5 


uoc u 1 tiuu lO'Visuai e^uiptnen t 


20*4 


8.0 


^-^ A i A u J. ulu lie vc j-opiflen t activities 


3^ •0 


17.9 




31 -7 


24-6 


TpAr'hPT/Q't'fl "Tf* ■nolo'f-trinQ 

1 cai^iic 1 / a ta 1 1 relations 


47*4 


28.5 


»juujcL. u luti u uci conipctcncy 


62o 


73*0 


i ^ pupj. 1 reiationa 


80.0 


76.9 


PI ATini T\0 RT\f\ T^T*^ T^A T*A +" 1 on 
L XCIililX IIQ CIIIIJ pi C pel 1 CI U X U 11 


f f • 1 


75.9 


Techniques of instruction 


77.3 


72.2 


Class discipline 


81 .5 


71 .2 


Innovative activites 


50. G 


41 .0 


Classroom management 


34.4 


75.4 


Student academic achievement 


33.8 


18.7 
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Teachers* experiences and opinions concerning actual and ideal criteria that should be 
used in evaluating teachers indicate that criteria related to classroom processes (e.g., 
techniques of instruction, teacher/pupil relations, classroom manage^nent) are both most 
prevalent in practice and most legitimate. As one moves away from this area towards criteria 
related to either the results of these activities (e.g., student academic performance or 
appearance of the classroom) or jroader involvement in educational activities (e.g., 
curriculum development activities, school and community activities, or teacher/administrator 
relations), one finds less frequent use of and less support for the criteria. On '.he whole, 
evaluation appears to be based for the most part on criteria teachers suppc^^ chough it is 
clear that it tends not to be based exclusively on these criteria. 

The pattern of responses of FLS teachers was very similar to that for ELS teachers, 
though all percentages for both actual and ideal criteria were somewhat higher, e.g., 81 per 
cent reported that involvement in the school and community is used as a criterion, and 37 per 
cent believe it should always be used as such. Classroom management was the major exception 
to this generalization — only 50 per cent reported its use as a criterion, and 64 per c^nt 
felt it should always be used. 

Standards 

While criteria (or, more accurately, criterion domains) define what is to be evaluated, 
standards are concerned with the adequacy of performance on a given criteria. For a very 
specific and easily measurable criterion, such as punctuality, standards are relatively easy 
to set. Fo' more general criteria, such as what constitutes adequate techniques of 
instruction, the sheer complexity of the process concerned may render impossible the full 
specification of standards. In such cases, standards are likely to be expressed in ways 
requiring considerable professional experience to interpret and apply, except in the most 
extreme cases. 

Teachers were asked how, in their view, standards were set in their last evaluation. 
They could check one or more of the following responses: by board policy or guideline; by 
the evaluator; by the evaluatee; collaboratively between the evaluator and evaluatee; don't 
know. Responses were consistent across different types of school and board. Thirty-eight per 
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cent indicated standards were set by board policy; 48 per cent by the evaluator; 6 per cent 
by the evaluatee; 10 per cent collaboratively; and 29 per cent did not know. PLS teachers 
responded somewhat differently: 63 per cent indicated that standards were set by policy and 
only 37 per cent by the evaluator. 

Data Collection 

Data collection is concerned with th^ sources of information on which evaluation is based, the 
types of information collected, who collected the information, and the time (and effort) spent 
collectinc tho information. 

Sources of Information 

The persons involved in providing information during the teachers' last formal evaluation and 
who teachers believe always ought to be involved in providing information are reported in 
table 36. Student involvement, though not high, is perhcps more prevalent than normally 
perceived. Interviews conducted with teachers and principals indicated that it is not unusual 
for principals to chat with children and review their work — thus, students provide 
information for the evaluation. As is seen later, it is rare for students to provide 
standardized typ»s of evaluative data on teacher performance. 
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Table 36: Actual and Ideal Persons Providing Teacher Evaluation Information, as <eported by 
Teachers 



Group Is Involved Ought Always Be 

Involved 

(n-3257) (n=4082) 



Toacher being evaluated 76. 0$ 87. 6^ 

Students 15*7 5»7 

Other teachers 2.3 3»4 

Parents 1.1 2.1 

School principal 38.1 75 ^9 

Other (e.g, superintendent) 3»9 27.0 



Clearly, teachers believe that they and their principals are and should be the primary 
sources of information for their evaluation, and that other teachers (such as department 
heads), students, or parents ought not be sources of data. There is support, though, for 
increasing the role of superintendents in this regard. 

Similar results to these were obtained for FLS teachers. 
Types of Information 

Information is as much characterized by how it is collected observation, collection of 
documents, interview, questionnaire, and so on as it is by its specific form. Tables 37 
and 38 report how information for the formal evaluation of teachers was collected and its 
specific nature, as well as how often teachers and, in the case jf specific types of 
information, principals and superintendents believe these n?ethods ought to be used. 
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J^^^^ ^^ i — Actual and Ideal Methods of Collecting Teacher Evaluation I nformation, as Reported 
by Teachers ■ — 



Actually Ought Always 

Used Be Used 

(n=3257) (n=40bc) 



Observation 


96. 1^ 


78.0 


Collection of documents 
(e.g. , lesson plans, 
student work) 


52.2 


52.6 


Interview 


48.0 


62.6 


Questionnaire 


4.5 


8.3 


Video taping 


0.7 


1 .2 



I^^^^.^Q- Actual and Ideal Types of Teacher Evaluation In formation, as Reported bv Teachers 

Principals, and Superintendents ^ 



Teachers 



Actual 



Ought 
Always 



(n=3257) (n=4082) 



Observation check- 
lists completed 

by observer 30. 4^ 32. 5^ 

Notes taken by 
observer on specific 
aspects of classroom 

activity 77.7 53.3 

General notes taken 

by observer 73.8 57.1 

Video tape of 
teaching and class- 
room activity 0.9 1 .0 

Questionnaires 
completed by 

students 0.5 1.8 

Self-evaluation 

questionnaire 5.7 25.7 

Self-evaluation 
written according 

to a guide 4,3 15.3 

Daily plan book 46. 0 27.1 

Standardized test 
scores for students 

in class 3.9 2.I 

Samples of student 

work 40.4 20.6 



Principals 

Ought 
Always 

(n»879) 



60.3 
54.6 

0.4 

0.6 

27.0 

18.0 
48.1 

2.1 
31.9 



Superin- 
tendents 

Ought 
Always 

(n-114) 



19.4!f 

70.5 
56.6 

0.0 

0.0 

25.7 

24.2 
52.3 

1.0 

32.4 
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Table 38 (continued) 



T "^hers 



Principals 



Actual Ought Ought 

Always Always 

(n»5257) (n-4082) (n-879) 



Results of teacher- 
made tests 8*5 



Written repct 
from previous 
evaluation 



7.0 



Student attendance 
records 3-4 

Record of interview(s) 
with evaluator(s) 16.8 



8.7 

8.9 
2.7 
19.7 



11 .8 

14.3 
2.0 
15.9 



Superin- 
tendents 

Ought 
Always 

(n«114) 



10.6 

31.8 
2.9 
37.4 



Table 37 suggests teachers would prefer more use of in\,arviews ana less of the collection 
of documents. Not evident in this table is the difference between elementary a.id secondary 
school teachers. For the former, documents were collected in two-thirds of the cases, and for 
the latter in only one-third of the cases. Responses for FLS teachers were similar, except 
that 15 per cent reported use of questionnaires, with almost a quarter believing that 
questionnaires ought always be used. 

Results for FLS teachers were similar with regard to the types of information used in 
evaluations at present; however, there was somewhat stronger support for the use of 
interviews (44.4 per cent) and of results from a previous evaluation (34.1 per cent). 

Taken together, the results in table 38 suggest a fairly strong consensus across the 
three levels of the hierarchy as to the appropriate types of information to be used. The 
results, too, are consistent with the views expressed as to the ideal criteria for formal 
evaluation, though our sample of items for the respondents to choose from may have been biased 
in favour of classroom activities and against school and community activities. In 
particular, there is little support of use of "output" criteria such as standardized test 
scores, and strong support for the use of observational notes. Perhaps one small surprise is 
the consensus that self-evaluation ought to be conducted much of the time, yet that in 
practice it is rarely conducted as part of the formal evaluation. 
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Collectors of Information 



Teachers were asked who was highly involved in carrying out their last formal evaluation. The 
figures were as follows, with unusual deviations among groups noted in parentheses: 
principal, 74 per cent; vice-principal, 25 per cent (18 per cent public elementary, 46 per 
cent public secondary, and 4 per cent separate); department head, 9 per cent (21 per cent 
secondary, less than 1 per cent elementary); co-ordinator , 2 per cent; superintendent, 16 
per cent (13 per cent public elementary, 26 pei cent separate elementary); director, 3 per 
cent; parents, 1 per cent; students, 6 per cent; and other, 6 per cent. The pattern for FLS 
teachers was similar, although 27 per cent reported involvement of their department head. 

Time Spent 

The time spent collecting information was assessed by questions concerning the number of 
observations during a formal evaluation, the frequency of post-observation conferences, and 
the percentage of their time that principals felt should be spent on teacher evaluation. 
Tables 39 and 40 report the data concerning the first two matters. 



l^^^^ — Actual and Ideal Number of Observations During Teacher Evalua tion as Reported by 
Teachers. Principals, and Superintendents 



Number of 
Observa- 


Teachers 


Principals 


Superin- 
tendents 


tions 

During 

Year 


Acr >ial 
(n=3215) 


Ideal 
(n-3946) 


Actual 
(n=79l) 


Ideal 
(n=858) 


Ideal 
(n«106) 


None 




5.9? 


0.4? 


1 .7% 


0.9? 


One 


48.2 


36.3 


5.7 


15.3 


21 .7 


Two 


23.0 


26.3 


17.8 


21 .1 


23.6 


Three/four 


14.5 


22.0 


37.2 


39.3 


35.8 


Over four 


8.3 


9.4 


36.3 


22.6 


17.9 



Not shown in these data are modest differences between elementary and secondary schools, 
e.g., 54 per cent of the secondary teachers reported a single observation while only 41 per 
cent of the public elementary and 49 per cent of the separate school teachers did so. 
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Secondary principals indicated, as well, that they thought fewer observations were necessary, 
e.g., 26 per cent supported a single observation whereas only 16 per cent of the public 
elementary and 9 per .ent of the separate principals did so. 

Table 39 suggests that both principals and superintendents have higher expectations for 
the number of observations than do teachers; at the same time, principals reported having 
conducted more observations during their last evaluations than teachers reported having 
received. 

The 91 FLS teachers reported a very different pattern of observations, with more 
experiencing the extremes: 15 per cent reported no observation, 23 per cent one, 19 per cent 
two, 17 per cent three or four, and 26 per cent more than four. Yet, less than 1 per cent 
believed no observations was acceptable and only 17 per cent believed more than four were 
required. Still, they tended to favour more observations than did the English-language 
sample. 

Of the 26 FLS principals, none reported having not conducted any observations as part of 
their last evaluation; 50 per cent reported more than four. Almost 60 per cent believed more 
than four observations were necessary. 

Experience and opinions regarding the frequency of post-observation conferences are 
reported in table 40. 

Table 40: Actual and Ideal Frequency of Post-Observation Conferences for T eacher Evaluation, 
as Reported by Teachers and Principals 



Teachers 



Principala 



Frequency 



After every 
Observation 

After most 
Obaervations 

After some 
Observations 

Never 



Actual Ideal 
(n-3257) (n-4037) 



47.75S 
9.3 

14.2 
19.5 



72.7$ 

14.4 

12.2 
0.7 



Actual 
;n-767) 



42.0$ 
29.9 

27.6 
0.5 



Ideal 
(n-87l) 



52.4$ 

31 .6 

15.8 
0.2 
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Althougn table 40 indicates relatively high agreement among teachers and principals as to 
principals who hold conferences after every observation, there is wide divergence in the 
percentages that report no post-observation conferences. As well, opinions diverge as to the 
necessity of post-observation conferences after every observation. Experiences and opinions 
of FLS teachers and principals were somewhat different. Relatively fewer (27 per cent) FLS 
teachers reported conferences after every observation, and this was confirmed by the 
principals' responses (21 per cent). However, while the opinions of FLS teachers as to what 
should be the case coincided with those of the ELS teachers, their principals' views did not; 
a majority of ELS principals felt that conferences after most observations were adequate. 

The average length of the post-observation conference was reported by teachers (mean of 
19 minutes, s.d. = 13) and principals (mean of 30 minutes, s.d. = 14). FLS teachers and 
principals both reported longer conferences: 37 minutes ( s.d. = 21) and 43 minutes ( s.d. = 
22) respectively. 

Principals were asked what percentage of their time they spent on matters related to 
teacher evaluation and what they ought to spend; superintendents were asked the latter 
question. Principals reported spending an average of 13 per cent of ( s.d. = 14) and felt they 
should spend 20 per cent (s^ = 14). Superintendents believed superintendents should spend 
about 20 per cent of their time (sjl = 15) on this matter. FLS principals reported spending 
19 per cent and stated that they would like to increase this to 25 per cent. 

Reporting and Fol low-Up 

After information concerning a teacher's performance is collected and analysed, a report is 
typically prepared which is shared w*th the teacher. Sometimes, plans are made to implement 
recommendations that emerge from the report. This section concerns the nature of the report 
and follow-up to it^ the destination of any report, who it is shared with and how, and 
follow-up activities that may or may not occur. 

Nature of Report 

Three features were taken to define the nature of the report: the presence or absence of a 
final conference at the end of the evaluation process, the form of the report that was 
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generated, and the presence or absence of an appeal process if the teacher believed the report 
to be inaccurate or unfair. 

Only 24 per cent of all ELS teachers reported having had a final conference. This 
relatively low figure should be interpreted in light of the relatively large number of 
teachers (48 per cent) who were observed only once and for whom a final conference separate 
from their post-observation conference was, by definition, impossible. When they were held, 
final conferences averaged 21 minutes in length ( s.d. = 14) not much longer than a regular 
post-observation conference. Results were comparable for FLS teachers, though, again, their 
conferences averaged about 10 minutes longer. 

The formal document summarizing a teacher's evaluation was a statement under several 
headings in 48 per cent of the cases; an unstructured statement, in 29 per cent; a form with 
ratings for various activities, in 16 per cent; -nd a form with a summary mark or score, in 6 
per cent. There was no formal document in 8 pe. ent of the cases, and some other type of 
document in 4 per cent. 

The format of the report varied somewhat among the types of school. Separate school 
teachers were slightly less likely to report use of all formats, and more likely to report no 
report at all (13 per cent). Secondary teachers were more likely than others to report the 
use of forms with ratings for various activities (21 per cent). FLS teachers were more likely 
than ELS teachers to indicate that they received a statement under several headings (57 per 
cent) or a form with a summary mark or score (37 per cent); they were less likely to report 
receipt of an unstructured statement (2 per c nt). 

Forty- three per cent of all ELS teachers reported that there was an appeal process 
available. Fourteen per cent reported no appeal route existed and 38 per cent did not know 
whether or not one existed. Secondary teachers were somewhat more likely than elem^-itary 
teachers to be uncertain about the existence of an appeal process (42 per cent vs. 35 per cent 
for public and 34 per cent for separate school teachers). Overall, FLS teachers responded to 
the various options in the same proportions as thei^* ELS counterparts. 
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Destination of Report 



Most teachers who had been evaluated indicated that they kept a copy of the report (84 per 
cent); a large proportion (43 per cent) Indicated a copy was sent to the board offices where 
it was maintained as a permanent record; and 23 per cent did not know what happened to the 
report. Other responses were: a copy was filed in the school but will be destroyed when the 
teacher leaves the school, 10 per cent; a copy is kept in the board offices and will be 
destroyed after a given number of years, 6 per cent; a copy is filed in the school on 
request and would otherwise have been destroyed, 2 per cent; the report was immediately 
destroyed and no copies were kept, 1 per cent; and other, 9 per cent. The pattern for FLS 
teachers was similar, although only 67 per cent reported having kept a copy. 

Report Sharing 

That reports were shared with teachers is evident from the high percentages (90 in the ELS 
survey and 95 in the FLS survey) who reported having signed their report. Very few (6 per 
cent) indicated they had not signed it and still fewer (4 per cent) did not recall. 

A series of questions concerning post-observation conferences provides some insight into 
the type of interaction that probably accompanied the presentation and discussion of these 
reports, particularly in view of the large percentage of teachers for whom there was only one 
observation and to whom the report would normally be presented at the post-observation 
conference. 

Both teachers and principals were asked about the freedom of participation in the 
conference. Seventy-eight per cent of the teachers indicated that both they and their 
evaluators participated freely; in 16 per cent of the cases, the evaluator was involved and 
the teacher was restrained; in 5 per cent both were restrained; and in 1 per cent the 
evaluator was restrained and the teacher was not. Principals' responses were very similar: 
83 per cent, 12 per cent, 5 per cent, and 1 per cent, respectively. FLS teachers and 
principals also had similar responses. 
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Similarly, most of the teachers (79 per cent) reported that the conference was not 
threatening; 19 per cent found it somewhat threatening; and only 2 per cent found it very 
threatening. 



All criticism offered during the conference was considered constructive by 57 per cent of 
the teachers; 25 per cent reported no criticism was given. Only 15 per cent felt the 
criticism was partly constructive and 4 per cent not at all constructive. 

Honest and sincere praise was reported by 87 per cent of the respondents, "^o only 8 per 
cent did the praise seem insincere and just 4 per cent reported no praise was given. 

At the end of the conference, 69 per cent of the teachers "felt good", 20 per cent felt 
neither positive nor negative, 8 per cent felt somewhat negative, and 3 per cent were uneas^ 
and defensive. Principals had similar, albeit slightly more positive, feelings: 72 per cent 
felt good, 24 per cent neither positive nor negative, 3 per cent felt somewhat negative, and 1 
per cent uneasy and disappointed. 

The reports by principals and teachers concerning their reactions to their conferences, 
which were similar in FLS and ELS sun/eys, imply that communication did take place as reports 
were shared and that, in most cases, the conference servad as an opportunitv to enhance the 
attituHss and commitment of the teachers. 

Fol low-Up 

After completion of a formal evaluation, there may or may not be a program to ensure that the 
conclusions of the evaluation are translated into recommendations and action. In most cases, 
there was no such program: only 14 per cent of all teachers (20 per cent in FLSs) reported 
that a plan was developed as a result of their last evaluation. Of those with plans, 43 per 
cent (58 per cent in FLSs) reported that the plans were monitored and 52 per cent (42 per cent 
in FLSs) that they had undertaken professional or academic activies to accomplish aims of the 
plan. 
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The relative lack of use of the results of evaluation for future undertakings was also 
reflected in the small percentages of evaluations that made use of earlier evaluation reports 
(table 38). 

Follow-through also took the form of actions resulting from an appraisal. Nine per cent 
reported receiving letters of commendation, 0.1 per cent being nominated for a teaching award, 
3 per cent being no longer under review, and 9 per cent receiving a permanent contract. Other 
beneficial results were reported by 8 per cent. Negative consequences also occurred: 0.2 per 
cent were placed under review and 0.2 per cent also failed to receive perman , contracts. 
Other negative effects were reported by 1.6 per cent. 

Actions affecting FLS teachers occurred with similar frequency, for the most part. 
Differencps included a smaller proportion being given permanent appointments (3 per cent), 
presumably because of the lack of growth in the number of positions in their boards, and a 
larger proportion (6.5 per cent) reporting other negative consequences. 

Evolution of Policy 

Scnool board policies generally develop over a n-imber of years, i, tally in response to a felt 
need that may ha/e a political or organizational basis. The rapid adoption of policies on 
personnel evaluation described in the preceding chapter is evidence that a need for such 
policies has been detected within most Ontario school boards. A series of questions sought to 
determine the formal process by which these policies were developed and implemented, the 
p'-ocesses various parties felt should be followed in these matters, and the degree of 
specificity in the policies developed. 

Implementation 

Implementation of a policy begins with its development; attitudes and knowledge formed during 
the process of developing a policy have a direct bearing on its use in practice. 

Teachers were asked who took part in the development of teacher evaluation policy in 
their board and who they believed ought to take part. A similar question was asked as to who 
ought to approve such matters. The results are reported in tible 41. 



ERIC 



135 15^ 



Table 41: Actual and Ideal Persons Involved in Developing and Approving Teacher Evaluation 
Policies, as Reported by Teachers 





DevelopmefAt 


Approval 


Group 


Ac tual 


Ideal 


Ideal 


(n-3048) 


(n-4082) 


(n-3048) 


Trustees 


7 •!% 


13. 3^ 


37. 8^ 


Di rector 


28.2 


38.9 


50.8 


Superintendents 


48.5 


70.8 


57.5 


Principals 


51 .8 


93.8 


54 .U 


Teachers 


28.6 


•70 1 
(O. / 


80*4 


Pa reuts 


0.8 


13.6 


NA 


Secondary students 


0.2 


8.2 


NA 


Outside consultants 2.8 


11.3 


NA 


Others 


4.8 


6.4 


10.1 



The general picture presented is one in which teacher policies are developed by "middle 
management" superintendents and principals with some involvement by teachers. Teachers 
generally support this approach, but would like to see more involvement of teachers. At the 
time of the survey, teachers -id not perceive trustees as having much involvement in the 
process and believed this situation to be appropriate. 

As far as the approval of guidelines was concerned, teachers clearly felt the teachers' 
federations should give formal approval; belief that it was also important for the trustees 
to do so was not strong. 

The perceptions and attitudes expressed were as true for teachers in separate as in 
public boards, as true at the elementary as at the secondary level, and as true in French- as 
in English- language schools. 

Qnze teacher evaluation prorpaures have been developed and approved, there may be a need 
for periodic reviews; the modal teacher believed such a review should occur about every fifth 
year. Only 22 per cent, however, indicated that their board carried out such a review on a 
regular basis; 11 per cent stated theirs did not and the majority, 67 per cent, were not 
sure. FLS responses did not vary significantly from those in other schools. 
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Degree of Specification 



The most basic level of specification is reflected in the existence or non-existence of a 
teacher evaluation policy. Seventy-five per cent of the teachers indicated that their board 
had procedures for formal teacher evaluation, 7 per cent stated theirs did not, and 16 per 
cent were not sure. Responses froc. FLS teachers in FLSs were virtually identical. 

The availability of supporting documents (e.g., a manual with directions on how to 
proceed, samples of appraisal forms) implies greater specificity of policy. Sixty-five per 
cent of the teachers in boards with policies reported that such documents existed, 9 per cent 
that they did not, and 26 per cent were not sure. Eighty-seven per cent of the principals 
indicated supporting documents existed, 12 per cent that they did not, and only 1 per cent 
were not sure. Still more superintendents, 96 per cent, indicated such documents were 
available in their boards. Just 3 per cent indicated they did not and 1 per cent were not 
sure. 

Responses from FLS teachers were similar to those from other teachers. All the FLS 
principals, however, stated that such documents existed. 

Teachers were asked to chack which of a list of possible problems with evaluation 
procedures applied in their school board. A number of the items bear on the issue of the 
specificity of the procedures. Overall, 16 per cent indicated that the "procedures lack 
detail", 17 per cent that they "do not help clarify roles", 16 per cent that "supporting 
documents are inadequate", and 20 per cent that the "procedures do not ensure fairness". 
However, 25 per cent felt the procedures "require excessive conformity to a single model of 
what it means to be a 'good' teacher", suggesting that in some regards policies may be too 
specific — or specific about criteria or standards about which substantial numbers of 
teachers disagree. 

Ab reported in the review of evaluation policies, various distinctions can be made in the 
form of policies, e.g., for permanent vs. probationary teachers, for summative vs. formative 
purposes, or for administrative vs. developmental purposes. Teachers, principals, and 
superintendents were asked whether or not they believed separate evaluation systems for 
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administrative purposes and for developmental purposes are pos«;ihlp. AUo, thpy were asked 
their preferences assuming such separate systems were feasible. Results are reported in table 
42. 

Table 42: Teacher, Principal > and Superintendent Opinions on the Feasibility and 
Desirability of Separate Teacher Evaluation Systems for Administrativt and Develo pmental 
Purposes 



Separate Systems Teachers Principals Superintendents 



Feasible 



(n-3974) (n=869) (n=109) 



Possible 44. 6^ 44.8? 57. 8? 

Not possible 24.7 33.8 27-5 



Not sure 



30.7 21.4 14.7 



Preferred System (n=3850) (n=837) (n=108) 



Strongly prefer 

separate systems 18.6? 20.4? 28.7^ 

Prefer 

separate systems 20.1 14.5 ^9*4 

No preference 27.6 13-4 9-3 

Prefer a single, 
multi-purpose 

system 2*5.0 33.7 26.9 

Strongly prefer 
a single, multi- 
purpose system 8.6 18.0 15«7 



Apparently, as one moves up the hierarchy, understanding concerns and opinions about this 
issue becomes more clearly defined. In the case of the feasibility of separate systems, the 
"not sure" response dropped from 31 per cent for teachers to 15 per cent for superintendents. 
And, while opinion is split on the matter, a clear plurality believes separate systems are 
feasible. 

But are separate systems desirable? Twenty-eight per cent of the teachers have no 
preference, while only 9 per cent of the superintendents straddle the fence. Overall, one 
would have to say the profession is split on the issue at all levels, though slight 
pluralities of teachers and superintendents prefer separate systems and a slight majority of 
principals favour a single system. 
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Impact of Policy and Practice 



One issue of considerable concern regarding any law, policy, or set of guidelines is the 
extent of compliance with it by those to whom it is meant to apply. Without compliance, a 
policy can have at most symbolic meaning and, in terms of ensuring effectiveness by guidinr 
individuals' actions, it will be a failure. A number of strategies exist for ensuring 
compliance; underlying all of them is the need for the exertion of effort by those within the 
organization (or society) responsible for implementing regulations. The success of their 
efforts, then, can be judged by the nature and degree of impact the procedures have on those 
whose compliance is sought. 

Thus, in this section, a chain with three links is considered: degree of compliance in 
terms of individuals' formal behaviour; the extent of effort expended to ensure compliance; 
and the impact evaluation policies have in terms of achieving the results intended by them. 

Degree of Compliance 

Teachers were asked to indicate whether or not teacher evaluation as practised in their board 
was very different from policy, approximately as in policy, as described in policy, or they 
were not sure. Their responses indicated that, for the most part, evaluation was practised as 
in policy (25 per cent) or approximately as in policy (41 per cent). Only 3 per cent felt 
practice was very different from policy, though 32 per cent were not sure. FLS teachers had 
similar experiences, with 32 per cent indicating practice followed policy, 37 per cent 
approximately followed, and 2 per cent did not follow. Twenty-nine per cent were not sure. 

The screening questionnaire also provided data regarding the degree of compliance with 
.iacher evaluation policies. 

Extent of Effort 

The effort committed to make the evaluation system work was assessed by a number of questions, 
some of which were primarily concerned with the effort the board made to implement the policy, 
and others which were concerned with the effort expended by the evaluators -- and evaluatees 
-- in carrying out a given formal evaluation. 
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Whether or not the board organized workshops or professional development days to 
familiarize staff with the policy was a key indicator of effort. Twenty-nine per cent of the 
teachers reported such workshops. Of those reporting workshops, 70 per cent had participated. 
In the workshops, attention was given to a description of the evaluation procedures (90 per 
cent), an explanation of the philosophy underlying the procedures (72 per cent), and a 
description of the purposes of the evaluation (79 per cent). Group discussion occurred in 50 
per cent of the cases and role playing in 22 per cent. Three per c^nt of the teachers rated 
the workshops very good, 28 per cent good, 57 per cent average, 8 per cent poor, and 3 per 
cent very poor. 

FLS teachers were more likely to be in boards with workshops (39 per cent) and to have 
participated (82 per cent). In these workshops, role playing was less likely (13 per cent) 
and group discussion more likely (69 per cent). Their ratings of the workshops were about the 
same as those of ELS teachers, though 9 per cent reported them as being very good. 

Workshops also were held for principals, of whom 49 per cent reported very thorough 
workshops and 29 per cent not thorough. However, 20 per cent reported no workshops at all. 
Principals were also asked what priority they attached to implementing the teacher evaluation 
policy in their board. Fifty per cent placed high priority on it, 46 per cent moderate 
priority, and only 4 per cent low or very low priority. 

Superintendents were asked questions identical to those asked principals. Their 
responses concerning workshops were virtually the same, but the priority they placed on 
implementing teacher evaluation policy differed. Seventy-five per cent placed high priority 
on it. 22 per cent moderate priority, and only 3 per cent low or very low priority. 

Responses of FLS principals were similar to those of other principals. Their priorities, 
however, were similar to t! ose of superintendents, i.e., they tended to place high priority on 
implementing policy. 

Effort is also reflected in the care with which individuals read the supporting documents 
to the teacher policy, "^hirty-four per cent of the teachers reported having read them with 
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wre, 31 per cent having read them casually, and 15 per cert not at all. Principals 
apparently expended greater effort in this regard: 84 per cent had read them carefully, 6 per 
cent casually, and 11 per cent not at all. 

If one were to summarize school boards' efforts to implement programs of teacher 
evaluation, one would conclude they have made moderate efforts to create and disseminate 
documents, and modest efforts at running effective workshops for administrative staff. 
Apparently, relatively little effort has been expended in ensuring that teachers are as well 
prepared to be evaluated as the evaluators are to conduct the evaluation. 

Board-wide efforts may or may not reflect the commitment and thoroughness with which 
individual evaluations are carried out. A number of items completed by teaclers and 
principals reveal the engagement of these parties in the evaluation process. 

The post-obse-vation conference is one key step in the evaluation process; the length of 
these conferencec and teachers' reactions to the sessions have already been reported. As 
well, teachers were asked how well planned these conferences were: 39 per cent responded very 
well planned, 51 per cent fairly well planned, and 11 per cent poorly planned. 

Similar questions were asked regarding the entire evaluation conference, i.e., how 
skilfully it was conducted and how seriously the process was taken by both of the parties 
involved. Twenty per cent reported their evaluators very skilful and 62 per cent skilful, 15 
per cent not very skilful, and 3 per cent not skilful at all. Fifty-five per cent took the 
process very seriously, 37 per cent somewhat seriously, and 8 per cent not at all seriously; 
in their perceptions, 60 per cent o.' the evaluators took the process very ceriously, 36 per 
cent somewhat seriously, and 5 per cent not at all seriously. Overall, elementary teachers, 
p-blic and separate, were more likely to check "very seriously" for both parties, as were FLS 
teachers. For example, 67 per cent of the latter group indicated this response. 

It would appear, then, that the individuals "nvolved - the teacher and the principal -- 
invest more effort in the process than might be expected given the level of effort expended to 
implement these policies on a board-wide basis. At the same time, there is evidence that this 
effort may not be as great ab it could be. 
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Nature and Degree of Impact 



Of greatest interest is the impact that performance appraisal has on the individual teache \ 
This impact can be divided into two stages: (1) changes in a teacher's knowledge, skills, and 
attitudes and (2) changes in a teacher's performance as a teacher. Effects at the first 
stage can be considered intervening effects that are prerequisite for changes in the second. 

While the general impact of evaluation systems is imporcant, it is perhaps more important 
to know what characteristics of a given system help to make it effective. Therefore, in 
addition to the description of the impact of teacher evaluation in Ontario as a whole, an 
analysis of the relationships among various characteristics of evaluation systems and the 
impact of these systems is presented in this section. 

Description of Effects 

Seven items, some of which have already been described in other sections, assessed the 
intervening effects of the evaluation process; these included both questions concerning the 
quality of the evaluation process (i.e., the teacher's satisfaction with the report form 
received, the teacher's perception of the fairness of the evaluator and the procedures, and 
the teacher's perception of the skilfulness of the evaluator) and questions about the 
teacher's attitudes towards the process of evaluation (i.e., the seriousness with which the 
process was taken by the teacher and the evaluator). 

Five items assessed the second stage impact of teacher evaluation, including three 
answered by teachers, one by superintendents, and one (on the screening questionnaire) by 
directors or their designates. Teachers were asked the extent to which the evaluation program 
was achieving the goals set for it by the school system, the extent to which it was achieving 
the personal goals set by teachers, and the extent to which their own performance improved as 
a result of their last evaluation. 

Most teachers were satisfied with the type of document used to summarize the results of 
their last evaluation. Specifically, 19 per cent were very satisfied and 66 per cent 
satisfied, while 10 per cent were dissatisfied and 5 per cent very dissatisfied. 
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Fairness was assessed on a 3-point scale and reveals a distribution rather similar to 
that for satisfaction with the report form. Eighty-seven per cent felt the judgement of the 
evaluator was fair and 82 per cent that the procedures were fair. As well, 8 per cent felt 
the judgement somewhat unfair and 5 per cent very unfair. Fifteen per cent believed the 
procedures were somewhat unfair and 4 per cent very unfair. 

Data on the other intervening variables were presented above in considering the effort 
individual teachers and principals had committed to the appraisal proce-s. 

The end results of the evaluation process, as far as this study is concerned, are 
measured by seven variables. In the screening questionnaire, directors were asked to rate the 
effectiveness of their evaluation system in achieving its goals. Of the 23 who responded, 30 
per cent rated their evaluation of teachers as very effective, 39 per cent as somewhat 
effective, and 9 per cent as ineffective. The remaining 22 per cent indicated the question 
was not appl icable. 

To a similar question, 24 per cent of the teachers responding indicated the evaluation 
system was achieving its goals either completely or to a large extent, 42 per cent to some 
exten , and 5 per cent not at all. As far as achieving teachers' personal goals, their 
assessment was somewhat less positive: 14 per cent responded completely or to a large extent, 
49 per cent to some extent, and 14 per cent not at all. In both cases, about 30 per cent of 
the teachers were not sure. 

Trustees were asked a similar pair of questions concerning the teacher evaluation process 
in their boards. With respect to administrative goals, the trustees' average response on a 
5-point scale (very low, low, moderate, high, and very high) was 3.0, implying a moderate 
level of success. They believed it was somewhat more successful in achieving its 
developmental goals, their responses averaging 3.2. 

Table 43 presents the responses of both teachers and superintendents as to the amount of 
improvement in teacher performance brought about by teacher evaluation. 
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Teachers and Superintendents 

Degree of Improvement Teachers superintendents 

(n-3158) (n-100) 

Not at all 40.5$ 0.0? 

A small amount 43.6 14.0 

A modest amount 13»3 43.0 

A aubstantial amount 2.6 43.0 

Elementary teachers reported that they were assisted somewhat more than did secondary 
teachers: 37 per cent of the former reported evaluation was of no help as opposed to 47 per 
cent of the latter. FLS teachers made a still more positive assessment: 9 per cent indicated 
that they improved a substantial amount, overcoming some major problems, 22 per cent reported 
modest improvement, 48 per cent a small improvement, and only 22 per cent indicated no 
improvement at all. Nevertheless, it is clear there is a wide disparity between the views of 
the teachers and the superintendents as to what the effects of teacher evaluation systems 
actually are on the evaluatees. 

Evaluation System Characteristics and Effectiveness . What, if anything, makes one evaluation 
system more effective than another? To answer this question, items under each of the major 
and minor c''<*gories of the framework used to analyse evaluation systems were correlated with 
a number of the intervening and resultant variables described above. Where consistent 
patterns of positive relationships occur, one can infer that the characteristics in question 
make for a more effective evaluatio*^ system. 



The appropriate level for this type of analysis is that of the organization, i.e., the 
school board. Therefore, the first step in the analysis was to aggregate individual level 
data for teachers and superintendents into board level scores. Specifically, for yes-no type 
questions the percentage of teachers or superintendents in the sample responding "yes" to a 
given item was calculated for each of the 30 school boards. For 3- and 4-point scales, the 
mean was calculated if there were relatively few missing or "not sure" responses (i.e. , 1 or 2 
per cent). If the number of missing or not sure responses was large (i.e., 15 or 20 per 
cent), then the percentage in one or two top categories (i.e., those responding "to a large 
extent" or "completely") was calculated. It was pot necessary to aggregate data for directors 
since there is but one director per school board. Finally, given the small number of boards 
in which data were collected for FLS teachers, it was necessary to limit the analysis to the 
results of the English-language questionnaire. 
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Methodologically, the AGGREGATE command of the SPSS-X statistical package was used to 
calculate the aggregate statistics, which were merged with board level data collected at the 
time of the screening questionnaire. Then, Pearson product-moment correlation coefficients 
were calculated (using the PEARSON CORR command) and the tables prepared. Results were 
considered to be statistically significant if their probability of occurrence und€'^ the null 
hypothesis (i.e., their g value) was less than 0.05. Given that the N was 30 for most 
correlations, any higher standard was viewed as too restrictive. To protect against 
overinterpreting occasional correlations that might be significant due to chance alone, an 
emphasis was placed on patterns, i.e., the existence cf more than one significant correlation 
for a given characteristic. As well, correlations that would be significant at the 0.01 level 
were noted. Alternative modes of analysis were considered (e.g., multiple regression, 
discriminant analysis, and path analysis), but were rejected because they provide results that 
are difficult for most people to interpret. 

The results of this analysis were organized according to the overall framework for 
analysis. Characteristics selected were those which either represented key stages in the 
evaluation process or seemed of particular interest. 

1. Preparation: Tables 44 through 47 display the correlations that various aspects of 
preparing for evaluation have with the intervening and final effects of evaluation. The first 
table relates the method of notification, pre-eval uation conferencing, and the use cf 
objectives-based evaluation to six intervening variables: teacher's satisfaction with the 
report form, fairness of the evaluator, fairness of the procedures, skilfulness of the 
evaluator, how seriously the teacher took the evaluation process, and how seriously the 
evaluator took it. Asterisks indicate significant Pearson product-moment correlation 
coefficients. 



Table 45 relates the same list of characteristics for planning for evaluation to the 
effectiveness criteria: director's rating of the effectiveness of the evaluation system, 
superintendent's perceptions of the extent of improvement brourht about in the typical 
teacher's performance as a result of evaluation, teacher's perceptions of the effectiveness of 
the evaluation system in achieving the school board's goals, teacher's perceptions of the 
effectiveness of the evaluation system in achieving teacher's personal goals, and extent of 
improvement teachers experienced as a result of their last evaluation. 
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Table 44: Relationship of Planning for Evdiuation to Intervening Effects of Teacher 
Evaluation 

" Intervening Effects^ 

Planning for 

Evaluation 



Item 45 55 56 57 58 59 

Mean 2.95 2.82 2.77 2.99 2.49 2.59 
3.d. .20 .06 .09 .12 .11 .11 



Method of Notification 

In person 53*4^ .33* .40* .25 .52** .17 -.08 
15.3 

Ey memo 29.4,? .26 -.13 '13 .08 -.06 .09 
14.2 

Visits 41.72 -.40* -.07 -.23 -.34* .07 -.02 

14.6 



Pre-evaluation Conference 

Conf. held 32. 5? .25 .14 .36* .27 .08 .17 

19.9 

Length 17.2 .30* .26 .27 .22 '-.04 -.19 

6.1 min. 



Objectives-based Evaluation 

23.7 
7.1 



Utilized 23. 7^ .21 -.08 .06 .32* .10 ,06 



Item 45 
Item 55 
Item 56 
Item 57 
Item 58 
Item 59 



Teacher satisfied with report form, 4 point scale. 

Fairness of evaluetor, 3 point scale. 

Fairness of procedures, 3 point scale. 

Skilfulness of evaluator, 4 point scale. 

How seriously teacher took evaluation, 3 point scale. 

How seriously evaluator took evaluation, 3 point scale. 



^ytSignif leant at the .05 level 
Significant at the .01 level 
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lapie KeiatlonsMp of Planning for Evaluation to Effectiveness of Teacher Evaluation 
(n = 30) 

Measurea of Effectiveness* No. of 

Planning for 

Evaluation Sig. r 

IteL> EFFTCH SUP60 13 14 60 

Mean 2 .28 3 . 25 24 .02 1 5 . 3? 1 .84 /Sign^ 
s-d- .67 .51 to. 2 7.5 .19 

Method of Notification 

In person 53.4? .13 .12 .04 -.03 -.09 3 ♦ 
15.3 

By memo 28. 4? .12 .03 .27 .30* .04 1 ♦ 
14.2 

Visits 41 .7^ -.15 .06 -.29 -.21 .05 2- 

14.6 

Pre-evaluation Conference 

Conf. held 32.5? .45* .18 .07 .13 .14 2 ♦ 
19.9 

Length 17.2 .26 .11 -.10 .00 .06 1 ♦ 

6.1 mm. 



Objectives-based E>^aluation 

23.7 
7.1 



Utilized 23.7? .46* .10 .24 .37* .41* 4 ♦ 



EFFTCH: Effectiveness of teacher evaluation - screening questionnaire, 3 point 
scale; n = 18. 

SUP60: Extent of improved teaching performance - superintendent 

questionnaire, 4 point scale; n = 23. 
Item 13: Achievement of evaluation goals for system, percentage of teachers 

indicating "completely'^ or "to a large extent". 
Item 14: Achievement of evaluation goals for teachers; percentage of teachers 

indicating "completely" or "to a large extent". 
Item 60: Extent of improvement in teacher's performance as a result of the last 

evaluation; 4 point scale. 

Number of significant correlations with each planning variable for both 
intervening (table 44) and final effect (table 45) variables. Signs indicate 
the direction of the correlations. 

Significant at the .05 level Significant at the .01 level 




In the righthand column of table 45 is a summary of the number of significant 
correlations positive ( + ) or negative (-) for each characteristic in both tables 44 and 45. 
For example, in the case of method of notification, notification in person exhibits three 
positive (3 +) correlations, by memor..r:dum one positive correlation (1 +), and by informal 
classroom visits two negative correlations (2 In addition, the tables contain the means, 

standard deviations, and sample sizes for each of the variables. A format identical to that 
used for tables 44 and 45 is used in the remaining tables or this analysis, with the taMes 
being presented in pairs. 

In the case of tables M and 45, as well as the other tables, a brief commentary is made 
emphasizing key points revealed in this analysis. The reader may wish to stud:, the tables in 
detail to discern more subtle relationships. 

Clearly, tables 44 and 45 indicate that the mode of planning for evaluation is related t-^ 
the effectiveness of the evaluation system. Method of notification seems primarily related to 
intervening effects such as the degree to which the evaluated teacher is satisfied with the 
form used in the evaluation and his or her assessment of the fairness and skilfulness of the 
evaluator. In particular, a personal meeting with the evaluatee before the process begins is 
positively correlated to these variables, whereas informal drop-in visiLs are negatively 
correlated with two of thpm. 

The positive correlations for pre-conferencing confirm the importance of this practice in 
the eyes, particularly, of directors. 

The use of objectives-based evaluation, of all the planning tools, has the strongest 
associations with the measures of effectiveness of the evaluation systems. Teachers are more 
likely to feel that their aspirations are being met by use of this approach and are more 
likely to report gains in performance as a result of the evaluation process. 

Purposes and criteria are dealt with in tables 46 and 47. Only two characteristics stand 
out as having a possible effect on the outcomes of teacher evaluation. The percentages of 
teachers reporting that the purposes of their evaluation were clearly given is correlated, in 
some cases strongly, with the dependent variables. Teachers' satisfaction with the report 
form, their assessment of the fairness of the procedures and the skill rf the evaluator, their 
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seriousness in the entire endeavour, their achievement of personal gca.s and improvement are 
all positively correlated with this characteristic, as are the directors' and superintendents' 
assessments of the effectiveness of the system. 

Ua!^?^Lclul!iun"'^''' ^""^ Criteria for F. a luation to Intervening Effects of 

(n = 30) 

Intervening Effects* 

Purposes and 

Criteria for ~ 



Evaluation 


Item 


45 


55 


56 


57 


58 


59 




Mean 
s.d • 


2.95 
.20 


2.82 
.06 


2.77 
.09 


2.99 
.12 


2.49 
.11 


2.59 
.11 


Purposes Clearly Given 














Yea 


70. 0^ 
1 1 .u 


.60 


.26 


.57 


.74 


.46 


.25 


Criteria Used 


in Evaluation 












Classrocm 
performance 


95.52 
7.1 


.82 


.48 


.63 


« 

.70 


.17 


-.12 


School 
coaisunity 


53. 0? 
19.2 


.15 


.11 


.08 


.^3 


-.12 


-.16 


Interpers . 
relations 


61 .7!? 
U.8 


.20 


-.03 


.10 


.1. 


-.Co 


-.19 


Prof, 
develop. 


45.8? 
17.0 


.22 


.09 


.15 


.26 


-.11 


-.27 


Personal 
qual. 


70.72 
8.5 


.13 


.20 


09 


.16 


-.03 


-.19 



Item 4b 
Item 55 
Item 56 
Item 57 
Item 58 
Item 59 



Teacher satisfied with report form, 4 point scale. 

Fairness of evaluator, 3 point scale. 

Fairness of procedures, 3 point scale. 

Skilfulness of evaluator, 4 point scale. 

How seriously teacher took evaluation, 3 point scale. 

How seriously evaluator took evaluation, 3 point scale. 



A^Signif icant at the .05 level 
Significant at the .01 level 
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Table 47: Relationship of Purposes and Criteria for Evaluation to Fffecti veness of Teacher 

Evaluation 

(n = 30) 

Measures of Effectiveness® 
No. of 

Purposes and 

Criteria for /Sign 
Evaluation Item EFFTCH SUP60 13 U 60 

Mean 2.28 5-25 24. 0| \5*'3% 1 .84 
t.d. .67 .51 10.2 7.5 -19 



Purposes Clearly Given 



Ye? 


7C.O^ 
1 1 .0 


«« 

.61 


« 

.35 


.16 


.34 


.38 


8* 


riteria Used 


in Evaluation 












Classi'oom 
performance 


95.5? 
7.1 


.33 


.13 


.01 


.00 


-.14 


. 4* 


School 
community 


53.02 
19.2 


.01 


.07 


.09 


.04 


.03 


0 


Interpers. 
relations 


61 .1% 
14.8 


-.13 


.15 


.06 


.02 


.17 


0 


Prof, 
develop. 


45. 8? 
17.0 


.03 


.03 


.00 


.12 


.12 


0 


Personal 
qual. 


70.7$ 
8.5 


-.08 


-.18 


-.02 


-.12 


..T6 


0 



^ EFFTCH: Effectiveness of teacher evaluation - screening questionnairt, 3 point 
scale; n = 18. 

SUP60: Extent of improved teaching performance - superintendent 

questionnaire, 4 point scale; n = 23. 
Item 13: Achievement of evaluation goals ^or system, percentage of teachers 

indicating "completely" or "to a large extent". 
Item 14: Achievement of evaluation goals for teachers; percentage of teachers 

indicating "completely" or "to a large extent". 
Item 60: Extent of improvement in teacher's performance as a result of the last 

t\/aluation; 4 point scale. 

^ Number of significant correlations with each planning variable for both 
intervening (table 44) and final effect (table 45) variables. Signs inaicate 
the direction of the correlations. 

Significant at the .05 level Significant at the .01 level 



^Iso, one criterion domain was positively rolated to several intervening variables. Use 
of classroom performance was positively related to satisfaction with the report form, fairness 
of the evciluator and procedures, and ski 1 fulness of the evaluator. With an average of about 
96 per cent of the teachers in a board reporting that classrooh p^^rformance was a criterion 
domain u^ed m their evaluation, this correlation is surprising and would appear to suggest 
omission of this domain undermines the legitimacy of the process as perceived by teachers. 
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Inclusion of criterion domains prevalent in comprehensive evaluations, however, had neither 
positive nor negative correlations with either intervening variables or measures o^ 
ef I ?ctiveness. 

2. Data Collection: The relationships of the sources and types of information used in 
evaluation to the effectiveness of the process are reported m tables 48 and 49, while the 
relationships for those who collected the information and time spent doing so are in tables 50 
and 51 The involvement of students in providing inrormation (a relatively rare occurrence) 
is positively correlated with directors', superintendents', and teachers' assessments of how 
well the evaluation system achieves its goals; however, it is not correlated with the effects 
on teachers as reported by teachers. 

(n - 30) 

Intervening Effects* 

oources and 



Types of 

Information Item 45 55 56 57 58 59 

Mean 2.95 2.82 2.77 2.99 2.49 2-59 

.20 .06 .09 =12 .i: .11 



Studer':s Provided Information 

Yes 16. 6^ .29 .06 -.16 .25 .00 -.02 

9.r 



Methods of Collecting Information 

Observation 96. 0? .72** .24 .31* .63** .21 
4.4 



.03 



Documents 53. 4? .02 -.07 -.09 .27 .29 23 
17.7 



Interview 45.6? .46 .26 .48 .65 .47 
(14.7) 



.36 
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Question- 4.4^ .03 .13 .21 .08 -.19 -.12 
nairea (4.7) 

Specific Types of Information Used 

Checklist 29. 9? -.04 .06 .22 .13 -.20 -.04 
16.7 

Specific 77.2? .51** .05 .05 .39* -.02 .03 
Notes 11.7 

Ceneil 73.8? .33* .20 .10 .50** .17 .14 

Notes 6.2 

Video 0.92 .02 -.04 .06 -.09 -.35* -.30 

3.2 

lb a 



Table 48 (continued) 



Student 


0-4$ 


.08 


.07 


.06 


• 24 


-•02 


-.08 


quest* 


0.9 














Self-eval* 


5 -955 


-.11 


* 

• 32 


• 23 


.21 


• 04 


-.08 


Ques • 


9.3 
✓ • ✓ 














Written 


3.5? 


-.04 


* 

• 35 


.26 


.22 


• 02 


-.16 


aelf-eval • 


5.9 














Daily 


50.7% 


-•20 


-.28 


- .22 


.06 




* 


plans 


19.2 














Standard 




.17 


-.02 


-.02 


.26 


• 03 


.07 


test scores 


^•2 














Student 


43-6^ 


.01 


-•12 


-.16 


• 16 


• 13 


.21 


wo rk 
















Teacner- 


8.1% 


.14 


-•09 


-.06 


• 20 


• 03 


. 10 


made tests 


8.7 














Prev. eval* 


S.9% 


• 18 


-•12 


-•18 


• 13 


-•08 


.08 


re port 


5-9 


















** 


* 




* 


* 


Student 




-.59 


-•33 


-•31 


-•41 


.07 


• 35 


at tend • 


4.5 














Interview 


16.1% 


• 03 


• 09 


» 


• 13 


-.04 


p 1 


record 


7.9 














^ Item 45: Teacher 


satisfied with 


report 


form, 


4 point 


scale. 



Item 55: Fairness of evaluator, 3 point scale. 

Item 56: Fairness of procedures, 3 point scale. 

Item 57: Skilfulness of evaluator, 4 point scale. 

Item 58: How seriously teacher took evaluation, 3 point scale. 

Item 59: How seriously evaluator took evaluation, 3 point scale. 

**Signif icant at the .05 level 
Significant at the .01 level 



Table 49: !^elationship of Sources and Types of Information Used to Effectiveness of Teacher 
Evaluation 
(n = 30) 

Measures of Effectiveness^ 

Sources and 



Types of 

Information Item EFFTCH SUP60 13 14 60 

Mean 2^28 3^25 ?4.0^ 15^3^ 1.84 

s^d^ ^67 .51 10^2 7^5 .19 



Students Provided Information 

Yea 16. 6? .39* ^49** •jJl* .27 -.03 3+ 

9^2 

Methods of Collecting Information 

Observation 96^0^ .03 .41 .35 .31 .12 5*- 
4.1 



No. of 
Sig. r 

/Sign^ 
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Table 49 (continued) 



Documents 


53. 4? 
17.7 


-.21 


-.06 


.10 


.16 


\ 

.32 


1 + 


Interview 


45.6? 
(14.7) 


.29 


.14 


.20 


.24 


« 

.34 


6> 


Question- 
naires 


4.4? 
(4.7) 


• 

.45 


— . \jy 




• 15 


-.19 


1 


Specific Type 


3 of Information 


US eu 










Checklist 


29.9? 
16.7 


.20 


-.19 


-.05 


• 15 


-.09 


0 


Specific 
Notes 


71.2% 
11 .7 




« 

.36 


« 

.39 


• 

•31 


.03 


5> 


General 
Notes 


73.8? 
8.2 


.37 


.27 


.20 


• 19 


.12 


2> 


Video 


0.9% 
3.2 


.29 - 


.06 


-.34 


-.09 


-.27 


1- 


Student 
quest. 


0.4? 
0.9 


-.15 - 


.18 


.01 


.13 


.18 


0 


Self-eval. 
ques . 


5.9? 
9.3 


.18 


.04 


-.18 


.06 


.06 




Written 
self-eval . 


3.5? 
5.9 


.02 


■13 


-.18 


-.07 


.11 




Daily 
plans 


50.7? 
19.2 


-.13 - 


.16 


.12 


.23 


• 

.39 


2+ 


Standard 
test scores 


3.5? 
3.2 


-.09 - 


.11 


« 

.35 


.14 


•03 




Stud ent 
work 


43.6? 
13.0 


-.27 - 


.16 


.05 


.03 


.22 


0 


Teacher- 
made tests 


8.7? 
8.7 


-.31 - 


.26 


.26 


.07 


.15 


0 


Prev. eval. 
report 


6.9? 
5.9 


-.31 


.11 


■36 


.26 


.05 




o tua ent 
attend . 


3.5? 
4.5 


-.25 - 


.38 


-.06 


-.19 


.14 


5- 


Interview 


16.1? 


.25 - 


.17 


.02 


-.02 


.22 


1^ 



record 7.9 
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EFFTCH: Effectiveness of teacher evaluation - screening questionnaire, 3 point 
scale; n = 18. 

SUP60: Extent of improved teaching performance - superintendent 

questionnaire, 4 point scale; n = 23. 
Item 13: Achievement of evaluation goals for system, percentage of teachers 

indicating "completely" or "to a large extent". 
Item 14: Achievement of evaluation goals for teachars; percentage of teachers 

indicating "completely" or "to a large extent". 
Item 60: Extent of improvement in teacher's performance as a result of the last 

evaluation; 4 point scale. 

^ Number of significant correlations with each pltinni-^g variable for both 
intervening (table 44) and final effect (table 45) variables. Signs indicate 
the direction of the correlations. 



Significant at the .05 level *^^i^ificant at the .01 level 



Table 50: Relationship of Who Collected Information and Time Spent on Evaluation to 



Intervening Effects of Teacher Evaluation 
(n = 30) 



Intervening Effects 



Who Collected 
Infonsatioa 



and Time 


Item 


45 


55 


56 


57 


58 


59 


Spent 














2.59 




Mean 


2.95 


2.82 


2.77 


2.99 


2.49 




s.d. 


.20 


.06 


.09 


.12 


.11 


.11 



Individuals Highly Involved in Process 

..15 -.02 



Principal 11.2% .07 
11 .0 



Supt. 



Students 



20. 1^ 
16.5 



..11 



15.55? -02 -.19 
5.3 



-.27 -.07 
•.32 



Number of Times Observed 

Number 2.67 .12 

0.40 



.20 



>13 



Length of Post-observation Conference 



Minutes 



18.7 
4.0 



.30 



19 



.33 



.09 
-.04 
-.14 

.35 
.40 



..12 .12 



*> «« 

.44 ^48 



.14 -.03 



.36 .25 



.30 .17 



Item 45 
Item 55 
Itrm 56 
Item 57 
Item 58 
Item 59 



Teacher satisfied with report form, 4 point scale. 

Fairness of evaluator, 3 point scale. 

Fairness of procedures, 3 point scale. 

Skilfulness of evaluator, 4 point scale. 

How seriously teacher took evaluation, 3 point scale. 

How seriously evaluator took evaluation, 3 point scale. 



^^Significant at the .05 level 
Significant at the .01 level 



I7j 
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^^^^^ — ^Relationshi p of Who Collected Information and Time Spent on Evaluation to 
Effectiveness of Teacher Evaluation 



(n = 30) 



Who Collected 

Information 

and Time item 

Spent 

Mean 
s.d. 



Measures of Effectiveness* 



EFFTCji SUP60 13 14 



60 



No. of 
Sig. r 
/Sign^ 



2.29 3-25 ?4.0^ 15. 3? 1.84 
.67 .51 10.2 7.5 .19 



Individuals Highly Involved in Process 



Principal 

Supt. 

Students 



77.22 -.09 
11 .0 

20. M -.11 
16.5 



15.52 
5.3 



.21 



Nujnber of Times Observed 



Number 



2.67 .16 
0.40 



.17 



-•33 



.07 



.21 



.04 



.04 



.20 



.00 



Length of Pos t-oosex va ixon Conference 

Minutes 18.7 -.13 .16 .22 
4.0 



.21 



.14 



,12 



.02 



,18 



.25 0 

.29 2* 
.22 2- 



.44 3^ 



.30 4^ 



EFFTCH: Effectiveness of teacher evaluation - screening questionnaire, 3 point 
scale; n = 18. 

SUP60: Extent of improved teaching performance - super intf?ndent 

questionnaire, 4 point scale; n = 23. 
Item 13: Achievement of evaluation goals for system, percentage of teacher: 

indicating "completely" or "to a large extent". 
Item 14: Achievement of evaluation goals for teachers; percentage of teachers 

indicating "completely" or "to a large extent". 
Item 60: Extent of improvement in teacher's performance as a result of the last 

evaluation; 4 point scale. 

^ Number of significant correlations with each planning variable for both 
intervening (table 44) and final effect (table 45) variables. Signs indicate 
the direction of the correlations. 

Significant at the .05 level Significant at the .01 level 
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Use of observation and, particularly, interviews is related to the number of intervening 
and end-result variables. The use of questionnaires and documents did not exhibit consistent 
relationships. 

Of all specific types of information, the use of specific notes stands out as having 
consistent positive relationships with the dependent variables, though general notes and daMy 
lesson plans exhibit two significant correlations each. On the negative side, use of student 
attendance records (a very rare practice) if negatively correlated with five of the 
intervening and output variables. The rate of usage of most other specific types of 
information seems unrelated to the effectiveness of the evaluation system. 

The degree of involvement reported, on average, for principals in the evaluation process 
appears to be unrelated to the effectiveness of the process, while the involvement of 
superintendents seems to make it a more serious affair, and that of students a less fair 
procedure (tables 50 and 51). The number of times observed and the iRngth of post-observation 
conferences also seem to matter. More observations appear to reflect a skilful evaluator, a 
serious process, and are likely to result in more assistance to the teacher. Longer 
conferences are associated with satisfaction, fairness, skilfulness, and the seriousness of 
the occasion, but appear not to be translated into more effective performance by teachers. 

3. R eporting and Fol low- Up: Tables 52 and 53 show the relationships between reporting, 
especially in the form of conferences and written statements, and follow-up with th-? various 
intervening ana dependent variables. 

Table 52: Relationship of Reporting and Fol low-Up to Intervening Effects of Teacher 

Evaluation 

(n = 30) 

a 

Intervening Effects 

Peporting and 

Follow-u? Item 45 55 56 57 58 59 

Mean 2.95 2.82 2.77 2.99 2.49 2.59 
s.d. .20 .06 .09 .12 .11 .11 

Conferences 

Conf. after 50. q? .18 .17 .25 -35* .12 .33* 
every obs. 16.5 

Final 24. 5? -.15 -.19 .03 .08 .21 .35* 

conf. 15.7 
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Table 52 (continued) 



Statements 



Under 42. 3^ 

headings 18.7 

Unstruct. 26.8% 
15.9 

Form w/ 13.6J 

ratings 16.5 

Form w/ A. 3% 
summary 

marks 6*7 

Appeal 39. 4i 

Process 17,5 

Follow-up 

Plan 16. 1? 

dev. 6.1 

Monitor 36. 2^ 

plan 19.6 

Activities 43. 1? 

18.4 

* Letter of 8.1? 

commend. 5-6 



.46** .05 .10 

-.06 .04 -.03 

-.05 .30* .25 

.05 .14 .13 

.30* .08 .23 

-.10 -.06 -.08 

.04 -.23 -.10 

-.17 -.02 .08 

-.02 -.06 -.15 



# 

.36 -.10 -.14 

-.29 .05 .00 

.05 -.11 -.13 

.04 .06 -.02 

.28 .22 .09 

.25 .25 .45' 

-.05 -.15 -.05 

-.15 -.01 -.19 

-.12 -.16 -.06 



Item 45 
Item 55 
Item 56 
Item 57 
Item 
Item 



58 
59 



Teacher satisfied with report form, 4 point scale. 

FairnGSS of evaluator, 3 point scale. 

Fairness of procedures, 3 point scale. 

Skilfulness of evaluator, 4 point scale. 

How seriously teacher took evaluation, 3 point scale. 

How seriously evaluator took evaluation, 3 point scale. 



**Signif icant at the .05 level 
Significant at the .01 level 
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Table 53: Relationship of Reporting and Fo11ow-Up to Effectiveness of Teac^^er Eva luation 
(n = 30) 



a 

Measures of Effectiveness 
Reporting and No. of 

^ 2ig.^r 

Foilow-up Item EFFTCH SUP60 13 U 60 /Sign 

Mean 2.28 3.25 24. Oi 15. 355 1.84 
s.d. .67 .51 10.2 7.5 .19 



Conferences 

Conf. after 50. 9? .33 -.01 .12 .09 .02 2* 

every obs. 16.5 

Final 24. 5J -.06 .06 .16 .20 .56** 2> 

conf. 15.7 

Statements 

Under 42.3? -.15 .21 .38* .24 .09 2* 

headings i6.7 

Unstruct. 26.8^ .20 .34 -.16 -.31* -.24 1* 
15.9 

Form w/ 13. 6? .24 -.28 -.13 -^12 -.38* U 

ratings 16. 5 ^" 

Form h/ 4.9? .45* .08 .14 -.06 -.42** 1^ 



s urjna ry 

marks 6.7 



1- 



Appeal 39.4? .34 .17 .31 .20 .10 2^ 

Process 17.5 

Follow-up 

Plan 16.1? .30 .20 .33* .37* .53** 4^ 

dev. 6.1 

Monitor 36.2? -.14 .44* -.03 -.29 .19 1^ 

plan 19.6 

Activities 48.1? -.37 .30 -.03 -.29 -.08 0 
18.4 

Letter of 8.1? -.01 .12 -.06 -.03 -.21 0 

comxaend. 5.6 



^ EFFTCH Effectiveness of teacher evaluation - screening questionnaire, 3 point scale; 

n 18. /, f 

SUP60: Extent of improved teaching performance - superintendent questionnaire, 4 point 

scale ; n = 23. • ^ * • 

Item 13. Achievement of evaluation goals for system, percentage of teachers indicating 

"completely" or "to a large extent". 
Item 14: Achievement of evaluation goals for teachers, percentage of teachers indicating 

"completely" or "to a large extent". 
Item 60: Extent of improvement in teacher's performance as a result ot the last evaluation, 

4 point scale 

^ Number of significant correlations with each planning variable for both intervening (table 
44) and final effect (table 45) variables. Signs indicate the direction of the 
O correlations. 

^Bll^ * Significant at the .05 level Significant at the .Olllc^O 



-T^ble 54: Relationship of E volution and Implementation of Policy to Intervening Effects of 

Teacher Evaluation 

(n = 30) 

Evolution and 

Impl ementat i on I nterveni nq Effects^ 



of Policy 


Item 45 


55 


56 


57 


58 


59 




Meen 2.95 
s.d. .20 


2.82 
.06 


C. 1 1 

.09 


c. yy 
.12 


o /in 

2. 49 
.11 


2. 59 
.11 


Participants in 


Pol icy Development 










Pri ncipals 


39.8% .55** 
17.2 


.02 


.36* 


.47* 


.12 


.16 


Teachers 


21.4% .36* 
17.3 


-.10 


.13 


.35* 


.21 


.17 


Efforts to Implement Policy 












Supporting 
documents 


57.0% .45** 
21.4 


-.06 


.17 


.33* 


.06 


.05 


How careful 
read? 3 pt. 
scale 


2.15% .08 
21.4 


-.07 


.11 


.10 


-.01 


.33* 


Held 

Workshops 


23.6% .22 
20.2 


-.15 




1 ? 

. LC 


m 




Partic. in 
workshops 
(if held) 


52.2% .33* 
22.7 


-.23 


.00 


.11 


-.01 


.10 


Fidelity of Implementation 












As in 
policy 


22.8% .25 
10.3 


-.07 


.07 


.31* 


.18 


.32* 



Item ^5 
Item 55 
Item 56 
Item 57 
Item 58 
Item 59 



Teacher satisfied with report form, ^ point scale. 

Fairness of evaluator, 3 point scale. 

Fairness of procedures, 3 point scale. 

Skilfulness of evaluator, 4 point scale. 

How seriously teacher took evaluation, 3 point scale. 

How seriously evaluator took evaluation, 3 point scale. 



)ic*Signif icant at the .05 level 
Significant at the .01 level 
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Table 55: Relationship of Evolution and Implementation of 
Policy to Effectiveness of Teacher Evaluation 
(n = 30) 



Evolution and 




Measures of Effectiveness 


« 




No. of 


Implementation 
of Policy 


Item 


EFFTCH 


SUP60 


13 


14 


60 


/Sign 




Mean 
s.d. 


2.28 
.67 


3,25 
,51 


24.0% 
10.2 


15.3% 
7. 5 


.,84 
. 19 




Participants in Policy Development 










Principals 


39.8% 
17.2 


.42* 


. to 


44** 


,46** 


. 25 


7+ 


Teachers 


21.4% 
17.3 


,15 


.53** 


.42** 


,47** 


,36* 


6+ 


Efforts to Implement Policy 












Supporting 
documents 


57.0% 
21.4 


.23 


.46* 


.20 


,22 


,25 


3+ 


How careful 
read? 3pt. 
scale 


2.15% 
.25 


,21 


-.10 


.25 


on 

. 20 






Held 

Workshops 


23.6% 
20.2 


,00 


.41* 


. 12 


,21 


.33* 


2+ 


Partic. in 
workshops 
(if held) 


52.2% 
22.7 


.45 


.50* 


.21 


,45** 


,25 


3+ 


Fidelity of Implementation 












As in 
policy 


22.8% 
10,3 


,42* 


.47** 


.61** 


,48** 


,15 


6+ 



17, 
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Table 55 (continued) 

Effectiveness of teacher evaluation - screening questionnaire, 3 point 
scale; n - 18. 

Extent of improved teaching performance - superintendent 
questionnaire, 4 point scale; n = 23. 

Achievement of evaluation goals for system, percentage of teachers 
indicating "completely" or "to a large extent". 

Achievement of evaluation goals for teachers; percentage of teachers 
indicating "completely" or "to a large extent". 

Extent of improvement in teacher's performance as a result of the last 
evaluation; 4 point scale. 

Number of significant correlations with each planning variable for both 
intervening (table 44) and final effect (table 45) variables. Signs indicate 
the direction of the correlations. 

. . ** 
Significant at the .05 level Significant at the .01 level 



EFFTCH: 
SUP60: 
Item 13: 
Item 14: 
Item 60: 
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The efforts made to implement policy, in the form of supporting documents, workshops, and 
the percentage of teac ^s who participated in workshops, oil exhibited two or three 
significant positive correlations with both intervening and dependent variables. 

Finally, the fidelity of implementation had six positive, significant correlations with 
the dependent variables. That is, boards where teachers reported that evaluation was 
practised as called for in policy were more likely to have systems rated as effective by 
directors, superintendents, and teachers alike. 

5. Demographic Variables : Though not falling within the framework used to analyse 
evaluation systems, characteristics of school boards such as size, rate of decline in 
enrolment, and teacher seniority could relate to the effectiveness of evaluation systems. 
W.n'ie not variables that school boards can normally control, they may provide clues to the 
types of board most likely to have effective systems. 

Tables 56 and 57 report analyses similar to the preceding 12 tables for five school board 
LharaCueristics (type, enrolment, year in which the policy was adopted or last revised, rate 
of enrolment decline, and percentage of French-language schools) end eight teacher 
characteristics (average age, per^^ntage with degrees, percentage who teach specialized 
subjects, mean years in same school, mean years in same board, mean years of expei .ence, 
percentage of staff who are male, and percentage with probationary appointments). 

Table 56: Relationship of Background Variables to Intervening Effects of Teacher Evaluation 
(n = 30) 



a 

Intervening Effects 

Bi.ckground 

Variables 

Item 45 55 56 57 58 59 

Mean 2.95 2.82 2.77 2.99 2.49 2.59 
a.d. .20 .06 .09 .12 .11 .11 

School System Characteristics 

Type 0.47 -.^0 -.23 -.09 -.03 -26 .35 

(0«pub; .51 
1«sep) 

1982 20,923 .19 .21 -23 .02 -.04 -.25 

Enrolment 24,906 
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Table 56 (continued) 



Year adopt 
policy 


197S 
4 


-.06 


-.15 


.10 


.13 


-.20 


-.26 


Rate of 
decline in 
enrolment 


2.22 
19.5 


-.10 


.05 


..08 


-.15 


. -/ ✓ 


« 

-.32 


Percentage 

French 

schools 


9.4? 
16.5 


-.08 


.10 


.15 


.07 


-.07 


.12 


sacher Characteristics 








• 




Average age 
10 pt-scale 


4.11 
.48 


.04 


.12 


.14 


.03 


-.08 


-.03 


Percent w/ 
degree 


69.8 
9.0 


.24 


.11 


.10 


.10 


-.22 


* 

-.38 


Percent w/ 
specializ. 


74.6 
9.7 


.16 


.34 


.06 


.06 


.14 


-.06 


Mean yrs 
in school 


7.23 
1.96 


-.09 


.12 


.00 


-.19 


## 

-.52 


-.37* 


Mean yrs 
in board 


11.2 
1 .8 


.10 


.21 


.27 


.10 


-.14 


-.30 


Mean yrs 
experience 


13.8 
1 .8 


.03 


. 20 


.23 


.07 


-.09 


. .13 


Percent 
males 


36.8 
12.2 


.05 


.17 


.04 


-.13 


-.28 


• 

-.33 


Percent on 
probation 


6.88 
5.14 


* 


-.25 


# 

-.31 


-.21 


.15 


• 

.34 



a 



Item 45 
Item 55 
I ten 56 
Item 57 
Item 58 
Item 59 



Teacher satisfied with report form, 4 point scale. 

Fairness of evaluator, 3 point scale. 

Fairness of procedures, 3 poir.t scale. 

Skilfulness of evaiuator, 4 point scale. 

How seriously teacher took evaluation, 3 point scale. 

How seriously evaluator took evaluation, 3 point scale. 



^^Significant at the .05 level 
Significant at the .01 level 
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Table 57 Relationship of BacKqround Variables to Effectiveness of Teacher Evaluation 
(n = 30) 



Keasures of Effectiveness 



a 



Background 



No. of 



Variables 














Sig. 




7 ten 


EFFTCH 


SUP^O 


13 


14 


60 


/Sign 




Mean 


2.26 


3.25 


24.02 


15.3? 


1 .84 






s.d. 


.67 


01 


1 0. 2 


n c 
7 . Z> 






School Systen 


Chf^racteristics 












Type 


C.47 


-.12 


-23 


-.03 


.20 


.41 




(C=pub; 


.51 














1 « jep) 
















1982 


20.923 


.09 


.09 


- .06 


-.19 


- . 15 


0 


Enrolment 


24.906 














Year adopt 

c 


10-^8 


— . ^ 1 


-.2 


-.01 


-.08 


-.03 


0 


policy 


4 














Rate of 


2 . ajo 




.14 


.03 


-.04 


-.17 


2- 


decline in 
















enrolment 


19.5 














Pe rcentage 


9.4^ 




-.13 


' .06 


.02 


.Ub 


U 


French 
















schools 


16. 5 














T'es'^^rr Characterist 


ICS 












Average age 


4.11 


.00 


.01 


.00 


-.18 


- .04 


0 


10 pt-scale 


.48 












- 






• 








* 




Percent w/ 


69.8 


.39 


-.17 


-.18 


-.15 


-.34 


1 ♦ 


degree 


9.0 












2- 








* 






• 




Percent w/ 


74.6 


-.15 


.34 


.03 


-.29 


-.31 


2* 


specializ . 


9.7 












1- 


Mean yrs 


7.23 


.12 


.19 


-.10 


-.24 


-.34 


3- 


in school 


1 .96 














Mean yrs 


1 1 .2 


.01 


-.21 


-.29 


* 

-.37 


-.12 


1- 


in board 


1 .8 














Mean yrs 


13.8 


.08 


-.16 


-.10 


-.20 


-.05 


0 


experience 


1 .8 














Percent 


36.8 


.12 


.10 


-.05 


-.27 


• 

-.46 


2- 


males 


12.2 


















* 












Percent on 


6.88 


..50 


-.11 


.17 


.11 


.14 





probation 5-14 
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Table 57 (continued) 



- screening questionnaire, 3 point 



^ EFFTCH: Effectiveness of teacher evaluation 
scale; n = 18. 

SUP60: Extent of improved teaching performance - superintendpnt 

questionnaire, 4 point scale; n = 23. 
Item 13: Achievement of evaluation goals for system, percentage of teachers 

indicating "completely" cr "to a large extent". 
Item 14: Achievement of evaluation goals for teachers; percentage of teachers 

indicating "completely" or "to a large extent". 
Item 60: Extent of improvement in L^acher's performance as a result of the last 

evaluation; 4 point scale. 

^ Number of significant correlations with each planning variable for both 
intervening (table 44) and final effect (table 45) variables. Signs indicate 
the direction of the correlations. 

Significant at the .05 level Significant at the .01 level 
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In interpreting these data, we must bear in mind that the variables are not independent 
of one another In particular, public boards of education are more likely to have more 
qualified staff and more males because they include a secondary panel and, historically, staff 
in secondary schools have been required to have degrees and have been more likely to be men. 
Ontario separate scfools, at this time, extend only to Grade 10. 

In terms of system characteristics, two variables appear related to the effectiveness of 
evaluation systems, type of board and rate of decline in enrolment. In the first case, 
separate school teachers report more effective evaluation systems; in the second, teachers in 
boards with higher rates of decline in enrolment (which, in the majority of cases, are public 
boards) report less effective systems. Size of board, year in which the evaluation policy wa", 
adopted, and percentage of French-language schools do not correlate with ef f ecti v/eness. 

Whether or not separate schools actually have more effective systems is difficult to say 
from these data. It will be recalled that elementary teachers, public or separate, reported 
that evaluation was more effective than did secondary school teachers. Thus, the observed 
correlation may come exclusively from the impact of the experiences of those in the secondary 
panels in public boards. A similar situation pertains in interpreting the relationship 
between declining enrolment and the effectiveness of evaluation systems. 

Several teacher characteristics are related to the perceived effectiveness of evaluation 
systems, including percentage with degrees (one positive and two negative), percentage with 
specialization (two positive and one negative), mean years in same school (three neodtive), 
mean years in board (one negative), pe^centr^ge male (two negative), and percentage on 
probation (one positive and three negative). Mean years of experience and average age showed 
no relationships. 

Interpreting these results is, again, difficult. One cluster of variables, including 
percentage v^ith a degree, percentage with a specialization, mean years in the same school, and 
mean years with the same board, all correlated with the characteristics of secondary teachers. 
It is impossible to determine which of these, if any, is a cause of the others. Are 
individuals with degi 2es less likely to profit f'^om evaluation because they are more skilled? 
Is the same true of those who teach specialized subjects? Are those with many years in the 
same school unable to profit from additional evaluations, possibly by the same principal? Are 
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males less likely to profit from evaluation because of male attitudes or becausj they are more 
likely to have degrees or because of other factors? It is quite likely that these questions 
are ut.answerable; in any case, they cannot be answered with tho data collected for this study. 

The percentage of teachers on probation also has correlations which are difficult to 
interpret. On the one hand, it is positively correlated with how seriously the evaluator took 
the process. This is logical; one would expect beginning teachers to perceive evaluators are 
serious. On the other hand, negative correlations with the fairness of the evaluator, 
skilfulness ot the evaluator, and directors' ratings of the system's effectiveness are more 
difficult to interpret. Perhaps the only logical conclusion is that, in boards that are still 
hiring many new teachers, evaluation tends to be a serious but not fully developed affair. 

Although virtually all these correlations are difficult to interpret, they do identify 
certain traits of teachers or school boards that are associated -- positively or negatively -- 
with the effectiveness of teacher evaluatio.T systems. It is apparent that either developing 
and implementing an effective system in a board with more highly educated staff with above 
average seniority is more d fficult than in other boards, or that such boards are less likely 
to work hard at the process. Ironically, boards at the other end of the spectrum, those with 
large percentages of probationary teachers, face a similar problem, though in these cases it 
is the quality of the evaluation e/oerience, rather than its effect on teaching, that is of 



concern. 



Performance Appraisal of Principals 

A sample of 1211 principals in 30 Ontario school boards — 16 public and 14 Roman Catholic 
separate -- was asked to reply to the English-language version of the principal questionnaire. 
Completed questionnaires were received from 879 principals, yielding an overall rate of return 
of 72.6 per cent. Of those returning questionnaires, 50.5 pet cent were from elementary 
public schools, 12.3 per cent from secondary public schools, and 36.5 per cent from separate 
schools; 0.7 per cent were not identified as to the type of school. Tcole 58 reports the 
distribution of respondents by region and type of school; column totals include those for whom 
type of school was not available. 
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Table 58: Distribution of Principals by Region and by iype of School 



Region Public Public Separate Total 

Elementary Secondary 





(n- 


444) 


(n. 


108) 


(n-321) 


(n. 


879) 




n 


% 


n 


i 


n 




n 


% 


Northern 


30 


6.8 


7 


6.5 


29 


9.1 


66 


7.7 


Western 


63 


14.2 


15 


13.9 


39 


12.1 


117 


13.3 


Central 


295 


66.4 


64 


59.3 


215 


67.0 


573 


65.8 


Eastern 


56 


12.6 


22 


20.4 


J8 


1 1 .8 


116 


13.2 


Total 


444 


50.5 


108 


12.3 


321 


36.5 


879 


100.0 



Of the principals who responded, b-1 per cent indicated that their board conducted formal 
appraisals of principals' performance. There was considerable variation in the responses 
between those in public elementary or secondary schools am* those ;n separate schools: over 
90 per cent of the former reported performance appraisals as compared with 69 percerit of the 
latter. 

Twenty-six principals returned French-language questionnaires. All were in Eastern 
Ontario and 89 per cent reported that performance reviews of principals occurred in their 
board. For the following analyses, responses are limited to those principals who reported 
that their board conr'L'cted formal principal evaluation. Most statistics are overall 
percentages and means; when differences among those in schools of different type or language 
are substantial, these are noted. 

Preparation 

Preparation for the performance appraisals involved four aspects: planning, purposes, 
criteria, and standards. 
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Planning 



For most principals, their last evaluation began with personal notification of the impending 
review followpd by both a request for materials and a pre-conference. Specifically, 58 per 
cent were notified personally, "^3 pe^ cent by memorandum, 33 per cent by informal visits to 
the school, 58 p.^r cent in a request for a statement about their school, and 25 per cent in a 
request for a self-evaluation. Secondary principals were more likely than others to have 
received a personal notice (76 per cent) and somewhat less likely to be asKed for a statement 
about their school (50 per cent), while FLS principals were more likely to experience informal 
visits (67 per cent) and to be asked for a self-evaluation (47 per cent). 

Pre-conferences were reported by 70 per cent of all principals; they were slightly more 
likely in separate school boards (recalling, of course, that this obtains only for those 
separate school boards with appraisal systems). Conferpnces averaged 51 minutes in length 
( s.d. ^ 27), and were slightly longer in separate school boards (61 minutes). 

As a result of the pre-conferences, 72 per cent of the principals reported understanding 
the appraisal procedures (S3 per cent of secondary principals and 60 per cent of FLS 
principals;. Sixty-eight per cent reported agreeing cn specific objectives during these 
conferences, though this p'^actice was apparently more common with separate school principals 
(82 per cent). 

Ovet.ill, 74 per cent of the principals reportt'd that objective-setLing was used in their 
last appraisal, whether or not they had a pre-cc iference. Twenty-one per cent indicated 
objectives were not set and the remaining 5 per cent eHher were not sure or did not answer. 
Objective-setting was reported with equal frequency among difft-rent types of scfool, but was 
less common in FLSs (53 per cent). 

When objective-setting was used, 47 per cent of the principals indicated that they set 
their own objectives and 51 per cent that the objectives were set collaboratively with their 
evaluators. Only 2 per cent reported that the appraisers set objectives for tnem. 




In a majority of the cases (76 per cent), objectives focussed on both the principal 
personally and the bChool as a whole; 19 per cent of the time the school was the focus, and 5 
per cent of the time the focus was on the principal personally. 



Purposes 



Principals were asked what the purposes of ^heir last appraisal were and what they believed 
the purposes of principal evaluation ought to be. Results are reported in table 59. 

Table 59: Actual and loeal Purposes of Principal Evaluation, as Reported by Principals 



Purpose Actually Used Always Ought 

to Be Used 

(n= S) (n=S79) 



To improve student learning 60- I? 77-3 

To develop the school community 59-4 44-0 
To identify administrative 

weaknesses in need of improvement 60.8 61 -9 

To identify instructional weaknesses 

in the school in need of improvement 43-7 61-5 
To cor ply with Ministry ^nd 

Board polioy 73-6 47-3 
To qualify pnncipctl for 

regular increment 1-1 6.8 

To select principals for promotion 5-9 24-1 

To identify inservice training needs 26-1 39-1 
To stimulate improved 

administrative performance 63-1 62.8 
To recommend first year principals 

for permanent assignment 6-7 44-4 

To identify principals for 
reassignment as teachers in case of 

school closings 1-6 12.2 

To establish evidence for demotion 
due to inadequate administrative 

performance 5-6 25-0 
To assess effectiveness of 

instructional program 57-0 58-6 

To ic'entify individuals for transfer 6-8 10-9 
To reassure and develop 

self-confidence 52-0 65-2 

Tvj as:-es3 and improve curriculum 51 -4 62-1 
To improve communication between 

principals and supervisory officers 52-7 55-7 

To clarify the principal's role 35-'' 55-0 
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m general, it appears that the purposes principal appraisal is seeking to achieve a^e 
those which have the principals' support. Primary among these are what can be termed 
developmental purposes, i.e., those related to the improvement of their on-the-job 
performance, including both administrative and curricular matter^ in the school. Evaluation 
appears rarely to be carried out for administrative purposes such as deciding salaries or who 
should be transferred, and principals tend to believe this is as it should be. One apparent 
exception to this general statement -- the small percentage reporting the use of appraisal in 
deciding whether or not an appointment should be made permanent contrasted with a considerably 
larger percentage believing this ought always be a purpose -- is explained by the relatively 
small percentage of principals surveyed who were new to the job and to whom this purpose could 
have applied. Only 7 per cent of thr principals surveyed were in the first year of a 
principal ship when last appraised. 



The purposes of their appraisal were effectively communicated to 76 per cant of the 
principals, they had not been made clear to 19 per cent, and 5 per cent were not sure. 
Effective communication of purposes was no more likely among those who had had pre-conferences 
than among those whu had not. 



Criteria 



Table 60 reports the criteria used in principals' last performance appraisal and those that 
principals, teachers, superintendents, and directors believe ought to be used. 
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Table 60: Actual and Ideal Critera Used in Principal Eva'.Jation. as Repo rted by Principals, 
Teachers, Superintendents, and Directors 



Criterion for Actual Ideal 

Appraisal Prin. Prin. Teach, Supt. Dir. 

(n-556) (n-879) (n*40e2) (r.-1U) [u-^S) 

General domains 



Administra tive 
perforsiance 


89. 7? 


84. 1* 


88.3% 


89*8% 


76. 0^ 


School and ccramunity 
relations 


87.8 


65.6 


64.8 


91 .7 


88. 5 


Prograo organization 


82,6 


77.8 


72.0 


89.8 


84.6 


Personnel ra£*nagement 


79.9 


79.9 


80.8 


88.9 


80.3 


Contribution to board 






18. 3 


■57 ,4 


42.3 


Contribution to 
religious education 


21 .2 


22.9 


12.7 


27.3 


44.0 


Routine administration 


of: 










Program 


76.6 


74.7 


71 .7 


90.7 


73.1 


Budget 


57.2 


52.2 


59.2 


73.1 


65.4 


Records 


50.2 


51.0 


55.7 


74.1 


57.7 


School plant 


52.7 


43.8 


41 .2 


61 .1 


53.8 


School discipline 


55.9 


57.9 


80. { 




7b .9 


Interpersonal relations 


with: 










Parents 


75.0 


64.8 


58.4 


82.4 


89.5 


Teachers 


76.8 


77.8 


82.0 


90.7 


92.3 


Other principals 


29." 


29.7 


26.1 


44.4 


40-0 


Supervisory officers 


43.9 


46.4 


37.1 


58.3 


60.0 


Improving school effectiveness through: 








Program development 


69.4 


71 .9 


64.9 


78-7 


88.5 


Program implementation 


71 .4 


74.8 


66.4 


89.8 


92.3 


Program evaluation 


66.5 


69.8 


61 .7 


84.3 


84.6 


Parent involvement 


54.9 


33.2 


32.1 


59.3 


57.7 


Staff evaluation 


71 .6 


60.7 


55.9 


66.1 


84.6 


Innovative activities 


49.3 


36.7 


38.7 


49.1 


53.8 


Assessing individu&l 
student needs 


4C.8 


54.9 


45.6 


66.3 


54.2 



18 

172 

ERIC 



Taken together, the results portrayed in table 60 suggest two points. First, both 
principals and those about them have very comprehensive views about the nature of the 
principal's ro^e; very few items score less than 50 per cent under the Ideal column, meaning 
that at least half of the respondents felt that virtually all of the items noted ought to be 
criteria in a principal's evaluation. Second, there is a high degree of consensus as to the 
appropriate criteria; on only a handful of items is there much divergence of opinion. Those 
exceptions are of interest, though. 

Among the general domains, principals express a desire for le^s emphasis on sc I and 
community relations, a direction with which the teachers concur Senior administrators, 
however, place strong emphasis on this domain. 

There also seems to be some contention as to the emphasis t^n should be placed on 
student discipline in a principal's appraisal. Principals feel that the current emphasis is 
about right, while teuchers and senior administrators seem to believe that more emphasis is 
called for. 

Also, in bringing about change in the school, particularly as it concerns program 
development, implementation, and evaluation, principals seeff zo prefer a slight increase in 
emphasis, teachers a slight decrease, and senior administrato-s a considerable increase. The 
pattern of opinions regarding the matter of staff evaluation s quite similar, though 
principals desire somewhat- less emphasis on this matter. 

It would appear, then, that in spite of a basic consensus as to what the principal's job 
is, there is some divergence of opinion as to what aspects should receive more or less 
attention in principals' performance appraisal. Teachers seom to be suggesting that the 
first priority is to maintain =^n orderly school environment, and the second is not to meddle 
in the classroom. Senior management seems to suggest that the first priority for principals 
is to bring about change by developing, implementing, and evaluating programs and by 
evaluating staff. 
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standards 



Taking criteria or criterion domains to represent what is to be evaluated, standards reflect 
the level of pf ^ormance that is expected. How are standards set in the evaluation of 
principals? Twelve per cent of the principals indicated that standards were set by board 
policie., 25 pt cent by the appraiser, 3 per cent by the appraisee, and 43 per cent 
collaboratively be'.ween the twc In addition, 17 per cent were not sure. 

Separate schoo principals were more likely to report that standards were set 
collaborative,/ (60 p-r cent) than were those elementary or secondary public schools, and 
less likely to report that they were set by policy or by the appraiser. In contrast, FLS 
principals were most likely to report standards were set by thf appraiser (53 per cent). 

Data Collection 

Four aspects of data collection were considered: sources of information, ' s of 
information, who collected information, and the time spent doing so 

Sources of Information 

Most principals (92 per cent) indicated that they had provided information to their 
evaluators. Their superintendent was the next most likely individual to provide information 
(40 per cent), followed by teachers (24 per cent), parents (7 per cent), students (7 per 
cent), and others (6 per cent). Students were somewhat -nore likely to provide information at 
the secondary level (15 per cent) than at the elemental level, publ c or separate. As well, 
FLS principals were more likely to report involvement of teachers (53 per cent) and their 
superintendents (80 per cent). 

In a companion question, teachers were asked who provided information in their 
principal's last appraisal. Nineteen per cent reported all teachers provided information and 
14 per cent that a few teachers did so. As well, 7 per cent reported information was sought 
from parents, 9 per cent from students, and 52 per cent from the area superintendents. 
Individually, 23 per cent of the teachers reported personally to have provided information. 
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confirming the reports of principals. FLS teacher responses were similar to those of ELS 
teachers, except that 81 per cent of those whose principals had been evaluated reported having 
provided information personally. 

Opinions concerning who ought to p^-vide information paralleled the actual situation. 
Almost all principals (96 per cent) thought that they always should provide information; as 
well, 89 per cent thought their superintendents should. Only 32 per cent thought teachers 
always should provide information, and 11 per cent that parents and students should do so. 
Teachers generally agreed with principals, except that 48 p^r cent thought that teachers 
always shoud be sources of data. Opinions of superintendents were similar to those of 
teachers, envisioning a somewhat greater role for teachers in the process than did the 
principals. 

Types of Information 

Reports of principals as tc the types of information actually used in their performance 
appraisals, along with their views and those of superintendents and directors as to the types 
that ought always be used, are presented in table 61. 



l^^^^ Actual an d Ideal Types of Information Used in Principal E valuatic- d Rtpurted by 

Principals, Superintendents, and Directors ^ ^ 

Type of Information Principala Supf s. Directors 



Actual 


Ideal 


Ideal 


Ideal 


( 


n-556) 


(n-879) 


(n-114) 


(n-26) 


Checklist completed 
by appraiser 


21 .6% 


15. 1? 


23.62 




Notes by appraiser 

on specific activities 


56.5 


45.9 


65.7 


65. 4 


General notes taken 
by appraiser 


56.8 


45.9 


62.3 


42.7 


Questionnaires completed 
by students 


K1 


2.8 


0.0 


0.0 


Self-evaluation 
questionnaires 


14.2 


49.9 


36.8 


41.7 


Written self-evaluations 


26.4 


52.1 


33.' 


32.0 


Daily calendar or diary 


9.5 


7.5 


16.2 


4.0 


Standardized test scores 
for students 


5.4 


2.1 


4.9 


0.0 


Samples of student work 


10. i 


8.0 


20.8 


0.0 


Grade reports of students 


6.7 


4.1 


12.4 


8.3 
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Table 61 (continued) 



Written report from 
previous appraisal 


16.2 


16.3 


36.7* 


57.7 


Student attendance 
records 


6.6 


2.7 


3.8 


4.2 


School budget 


27.5 


1 5-6 


38.3 




School timetable 


7C.9 


21 .9 


43-0 


32.0 


Student discipline 
records 


14.0 


9.2 


17.9 


28.0 


ocnooi nanCDOOK 






40.0 


44.0 


School newsletter 
to parents 


40.5 


23.9 


47.0 


24.0 


Coal package principal 
prepared for self 
and school 


70.0 


66.2 


61 .0 


92.3 



There appears to be a fair degree of consensus between principals, superintendents, and 
directors as to the types of information that ought to be used in principal appraisal, and for 
the most part thes? coincide with the types of information actual ^' used. In particular, 
there is strong support for the idea that principals should submit a "goal package" that they 
have prepared for themselves and the school. Also, there is agreement that notes, both 
specific and general, taken by observers should be used and that school output measures, such 
as standardized test scores or student absence records, should not. Nevertheless, there are 
moderate differences of opinion about the use of a number of types of info'^mation. 

First, as one goes up the hierarchy, one finds greater support for the use of checklists 
to be completed by the observers. Principals see this technique as being used too much at 
present, while superintendents and directors believe it should be used more. Second, 
supetintendents seem to have highest regard for certain "intermediate" products originating in 
the school (items such as newsletters, budgets, and timetables) while principals perhaps see 
the^e as used too much at present. Third, principals seem to place more faith in 
self-evaluation than do their supervisors. Finally, directors would like to see reports from 
the previous appraisal be uced as part of the information in a current appraisal. In summary, 
it appears that both of the parties involved the principals and their supervisors place 
greater omphasis on the information tney provide and less on that provided by the other. 
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Similar patterns of responses were evident in public elementary, separate, secondary, and 
FLS schools. A few minor deviations are worth noting, though. Secondary principals were more 
likely to report use of intermediate output measures such as student attendance records (16 
per cent), student discipline records (26 per cent), and a school handbook (37 per cent). As 
well, a greater proportion reported use of a previous evaluation report (30 per cent). FLS 
principals were far more likely to report use of student disciplinary records (73 per cent), 
and they tended to support regular use of this information in the review of principals (42 per 
cent). 



Collectors of Information 



Either a single superintendent or ^ team of superintendents was most likely to be involved in 
collecting information and carrying out a principal's performance appraisal. Ninety per cent 
of the principals reoorted the former approach was used in their last evaluation while the 
remaining 10 per cent reported a team was used. 

When asked who was highly involved in their last appraisal, the principals responded a;> 
follows: themselves, 82 per cent; teachers, 15 per cent; department heads, 2 per cent (15 
per cent in secondary schools); vice-principal, 8 per cent (17 per cent secondary); 
co-ordinators, 1 per cent; other principals, 1 per cent; their superintendent, 91 per cent; 
other superintendent(s), 14 per cent; the director, 7 per cent; parents, 2 per cent; and 
students, 1 per cent. 



Except as noted, the pattern was quite uniform across schools of different type. FLS 
principals, however, reported a somewhat different pattern. Only 57 per cent reported having 
been highly involved themselves and i.one reported use of other superintendents or an 
evaluation team. 



Time Spent 



Principals were asked how many days their evaluators spent collecting information, and how 
many they felt should be spent. Superintendents and directors were asked how many days they 
believed should be spent collecting information to evaluate a typical elementary school with 
300 pupils and 15 teachers. The responses are reported in table 62. 



Table 62: Actual and Ideal Amount of Time Involved in CoHectlnQ Princip al Evaluation 
Information, as Reported by Principals. Superintendents, and Directors 



Number of Days 


Principal 


Supt's. 


Directors 


Actual 


Ideal 


Ideal 


Ideal 




(n=537) 


(n«858) 


(n-108) 


(n-25) 


None 




1 .Bi 


0.0 


0.0 


One 


27.7 


10.7 


4.6 


4.0 


Two 


12.5 


16.5 


5.6 


4.0 


Three 


11 .5 


20.5 


24.1 


16.0 


Four 


7.3 


6.0 


13.0 


8.0 


Five 


6.9 


18.0 


22.2 


24.0 


Kore than five 


10.6 


26.5 


30.6 


44.0 



It would appear that actual and ideal practice diverge considerably as far as time spent 
collecting information is concerned. And, the amount of time considered necessary to evaluate 
a principal increases with a person's position in the hierarchy. 

The amount of time reportedly spent in collecting information was relatively uniform 
across different types of school, although secondary principals were somewhat more likely to 
report that their appraisers had spent more than five days (21 per cent). FLS principals 
reported a very different picture, however Of the 15 of them evaluated, 20 per cent 
indicated no days had been spent, while 47 per cent indicated more than five days; the 
remainder were spread out between these extremes. 

Superintendents additionally were asked what percentage of their time was spent on 
evaluation and what percentage they felt should be spent. The average superintendent 
reportedly spent 16 per cent of his or her time on evaluation ( s.d. = 15); the average 
preferred amount of time was 24 per cent (j^iA. ~ 1^)- 

Reporting and Fol low-Up 

Reporting and follow-up are concerned with both the nature of the report and the procedures 
followed, including where copies of the report are sent, how the report is shared, and the 
follow-up that occurs once the formal performance appraisal is completed. 



178 190 



ERLC 



Nature of Report 



To conclude an evaluation, a written report is normally prepared. Often, this is discussed at 
a post-conference and plans may be made for acting on recommendations made in the report. If 
a principal is dissati :>f ied with the results, an appeal process may be available. 

In 83 per cent of the cases, principals reported that a post-conferencc was held. 
Ninety-seven per cent of the principals repor^ed that their superintendents were at this 
conference; 12 per cent reported that another supervi.,ory officer was also present. These 
conferences averaged 53 minutes in length ( s. d. = 35). 

Most principals reported that post-conferences were very well planned (33 per cent) or 
fairly well planned (54 per cent). Typically, both parties participated freely (86 per cent), 
though 9 per cent reported they had been restrained and their evaluator had not, and 4 per 
cent reported both were restrained. 

Eighty- three per cent of the principals reported the nost-conference was nu*, at all 
threatening, 15 per cent somewhat threatening, and 2 per cent very threatening. Criticism 
tended to be completely constructive (67 per cent); 12 per cent reported it was partly 
constructive, and 19 per cent that it was altogether absent. Only 2 per cent reported 
entirely unconstructive criticism. 

Most principals felt good after the post-conference (70 per ce.it); 20 per cent felt 
neither good nor bad. Eight per cent reported feeling somewhat negative and 3 per cent very 
uneasy and defensive. 

The statement principals received tended to be either a statement under several headings 
(48 per cent) or an unstructured statement (21 per cent). Thirteen per cent reported a form 
with various ratings and only 5 per cent a summary mark or score. Eighteen per cent reprrted 
there was no formal document. Of those receiving statements, 20 per cent were very satisfied 
with the type of document, 62 per cent satisfied, 11 per cent dissatisfied, and 8 per cent 
very dissatisfied. 
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Thirty-six per cent of the principals stated there was an appeal process they could have 
used had they b-'^^r dissatisfied with the results of their appraisal, 25 per cent stated no 
such process existed, 26 per cent were unsure, and 12 per cent did not answer the question. 
Of the 194 principals responding, 1 per cent indicated they had appealed. This represented 
appeals by two principals, both in public elementary schools. 



Destination of Report 



Eighty-four per cent of the principals reported that they kept a copy of their evaluation 
report; 70 per cent indicated a copy also went to their school board's office as part of 
permanent record. Other responses were rare: 3 per cent that a copy was placed on file i 
the school and will be destroyed when the principal leaves; 8 per cent that a copy is kept 
the board's offices and will be destroyed after a certain number of years; and 8 per cent 
uncertain what happened to copies of the report. 



Report Sharing 

Seventy-two per cent of the principals reported that they were required to sign their 
evaluation report, 24 per cent that they were not. and 4 per cent did not recall. Whether oi 
not the report was shared with others was not determined in the questionnaire survey, though 
it was clear from the case studies that, in boards conducting principal appraisals, principa 
are usually requested to share the results, as appropriate, with their staff. A few boards 
beyond this and evaluators report to the school staff concerning their impressions of the 
school, omitting any evaluative comments concerning the principal. 



Fol low-Up 



A plan of action was developed for 28 per cent of the principals responding; such plans we 
most common at the secondary level (41 per cent) and less common in FLSs (7 per cent). Of 
those with plans. 59 per cent reported that the plans were monitored and 66 per cent report 
having engaged in academic or professional development activities to carry out their plans. 
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other types of follow-up occurred on occasion. Seventeen per cent of the principals 
reported letters of commendation; 1 per cent (representing five of the 556 principals 
responding) v/ere placed under review. 

Evolution of Policy 

The process of developing and implementing a policy for principal appraisal and the specifics 
of the policy were considered two key aspects of the policy's evolution. 

Implementation 

Principals reported rather widespread involvement of others in the development of their 
board's policies. Almost always involved were superintendents (94 per cent) and principals 
(74 per cent). Other individuals or groups involved were directors (64 per cent), trustees 
(20 per cent), teachers (15 per cent), consultants (8 per cent), and parents (2 per cent). 
Principals generally supported this pattern of involvement, though 65 per cent saw greater 
teacher involvement as desirable. 

Once procedures were developed, over 80 per cent felt the director, superintendents, and 
principals should be involved in granting formal approval to them. Only 65 per cent thought 
that the trustees need be involved in this and just 59 per cent that the teachers' federations 
should be requested to approve them. 

Workshops to assist in the implementation of the policies were reported by 57 per cent of 
the principals. Half of these principals rated the workshops as very thorough, and the others 
as not being thorough. However, 71 per cent of the directors reported having had thorough 
workshops in their board; 50 per cent of the superintendents reported thorough workshops and 
26 per cent workshops that were not very thorough. 

Degree of Specification 

As noted earlier, 84 per cent of che principals indicated that some form of principal 
appraisal was carried out in their board. Seventy-three per cent indicated that there were 




documents that described the criteria, procedures, and so forth, ani 98 per cent of the 
superintendents and 94 per cent of the directors indicated their board Sad such documents. 



There were problems with current procedures in the eyes of many principals. Twenty-seven 
per cent felt that the policies lacked sufficient detail; 20 per cent that the procedures did 
net clarify roles adequately; and 27 per cent that the support documents were inadequate. In 
short, a significant number of principals perceive policies as being not sufficiently 



Questions were not asked concerning the separation of evaluation for administrative 
purposes from developmental purposes. From data on the purposes of evaluation, however, it is 
clear that principal evaluation is primarily concerned with improving a principal's 
performance on the job and not administrative matters. In general, the separation of 
evaluation for different objectives did not appear to be as significant an issue for principal 
evaluation as it was for teacher evaluation. 

Impact of Policy and Practice 

The effects of policies for the appraisal o^" principals depend upon the degree of compliance 
with policy and the efforts made in implementing and administering Dolicy. In this section, a 
description of the findings in each of three areas, compliance, effort, and impact, is 
presented, followed by an analysis of the relationships of various features of performance 
appraisal policies with different types of impact. 

Degree of Compliance 

Principals, superintendents, and directors were asked the extent to which the practice of 
principal evaluation followed the policy. Their assessments are reported in table 63. Most 
felt that policies were followed relatively well. 



specific. 
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Table 63: Degree of Fide lity or Practice to Policy in Principal Evaluation, as Reported by 
Principals, Super intendents > and Directors 



Degree of Fidelity Principals Superintendents Directors 



(n=879) (n=89) (n=17) 

Very different from policy 5.0^ 0.0 0.0 

Approximately as in policy 

or as in policy 72.5 88.8 76.5 

Not sure n .5 3.4 

Not applicable 8.1 7.9 1 1 .8 



Extent of Effort 



A number of items, some already Hogcribed, indicate the effort put into implementing principal 
appraisal. Workshops and their quality are two indicators; another is the amount of 
resources committed to the program. 

Sixteen per cent of the principals believed a great deal of resources had been committed 
by their board to implementing the program, 28 per cent not much, and 38 per cent little or 
none. As well, 17 per cent were not certain. 



Directors and superintendents were asked the priority they placed on implementing 
principal evaluation in their jurisdiction. Eighty-two per cent of the directors and 80 per 
cent of the superintendents indicated it was a high priority; 18 per cent of the former and 
17 per cent of the latter placed moderate priority on it. Low or very low prioity was given 
it by 3 per cent of the superintendents. 



Effort is also reflected in the skin with which the process is carried out. Principals 
were Quite positive on this matter, 20 per cent indicating that their evaluators were very 
skilful and 63 per cent that they were skilful. Only 15 per cent found them not very skilful 
and 3 per cent altogether lacking in the needed skills. 

Finally, the seriousness with which the process is taken is an indicator of effort; if 
one is not serious about an endeavour, little effort is likely to be forthcoming. Fifty-eight 
per cent of the principals indicated that they took their evaluations very seriously and 59 
per cent believed their appraisers did so. As well, 35 per cent of the principals took them 
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somewhat seriously and 7 per cent not seriously at all. As for their appraisers, 59 per ce 
of the principals believed that they had taken it very seriously. 37 per cent .omewhat 
seriously, and 3 per cent not seriously at all. 

In sum, one would conclude that moderate efforts have been made at implementing apprai 
processes for principals. 



Nature and Degree of Impact 



Impact can occur in the form of a change in a person's internal self or his/her external 
behaviour. Information was sought on both aspects of principals' reactions to appraisal. 

Their reactions to post-conferences have already been described. It seems fair to 
conclude that, in the majority of cases, principal evaluation serves to reinforce the 
commitment and morale of principals. Many leave the process feeling very good about it and 
themselves. 

A positive response may in part depend upon the perceived fairness of the process. 
Eighty-six per cent of the principals felt the judgement about their performance was fair; 
only 8 per cent felt it somewhat unfair and 6 per cent very unfair. Responses concerning the 
procedures were slightly less positive: 80 per cent judged them fair, 15 per cent somewhat 
unfair, and 5 per cent very unfair. 

Trustees saw principal appraisal as being moderately successiul (3.2 on a 5-point scale) 
in achieving its administrative objectives. Their average response was marginally lower (3.1) 
for the achievement of developmental objectives. 

Finally, principals, teachers, and directors were asked how much improvement resulted 
from principal evaluations. Both principals and teachers were asked to reflect on the 
situation in which they were last involved. Their responses are in table 64. 
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l^^.^^.^^- — Degree of Improvement in Principal Performance After Ev aluation as Reported by 
Principals. Teachers, and DirectX ' — ^ 



Degree of Improvement 


Principals 


Teachers 


Directors 




Kn'523) 


(n«982) 


(n*16) 


Not at all 


18. 5« 




0.0? 


A small amount 


51.8 


13.0 


12.5 


A modest amount 


24.7 


9.9 


50.0 


A substantial amount 


5.0 


1.9 


37.5 


Don't know 




49.1 





It IS clear from these results that directors see principal appraisal as having more 
positive re ults than do the other two groups queried. At the same time, there seems to be a 
consensus that the practice does have an effect, which, in some cases, may be substantial. 

[valuation System Cha racteristics and Effectiveness . To determine if specific features of 
principal appraisal systems are related to tne results just reported, we carried out a 
correlational analysis similar to that conducted for teachers. The results are presented in 
tables 65 through 78. 

1. Preparation: The relationships of different dimensions of planning for principal 
appraisal and of the purposes ^^nd criteria used in the process with various measures of the 
effect of appraisal are reported in tables 65 through 68. As far as planning is concerned, 
only one category stands out: pre-conferencing. The existence of a pre-conf erence and, 
particularly, its length show a number of positive correlations with several intervening or 
final-effect variables. Results for other characteristics, such as the method of notification 
or use of objecti V3s-based evaluation, are either mixed or negligible. 
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Table 65: Relationship of Planning for Evaluation to Interveni ng Effects of Principal 
Ev aluation 



Intervening Effects^ 



planning for Item 35 45 46 47 48 49 

Evaluation Mean 2.39 2.79 2.76 2.99 2.56 2.61 
3.d. .35 .15 .16 .24 .30 .27 



Method of Notification 

In person 56.6? .01 .15 .10 -.09 .29 .27 
21 .0 

By memo 27. 6^ .47** .21 .23 .19 .05 .14 
25.8 

Visits 36. 6? -.38* .17 .19 .27 -.08 -.07 

21.2 

Statement 58. 6^ .16 -.13 -.08 -.18 -.60 -.37 
on school 25.4 

Pre-evaluation conference 

Conf. held 68.0% .34* .31 -34* .24 -.07 -.08 
25.8 

Length 48.6 .45 .53 -59 -57 .32 .36 

15.3 min. 

Objectives-based 

Utilized 73. 2? .45 .03 .17 .00 -.34 -.15 
30.3 



a 



The n may vary among correlations due to missing data. 



^ Item 35: Principal satisfaction with report form, 4 point 
scale. 

Item 45: Fairness of appraiser's judgement, 3 point scale. 

Item 46: Fairness of procedures, 3 point scale. 

Item 47: Skilfulness of appraiser, 4 point scale. 

Item 48: How seriously principal took appraisal, 3 point scale. 

Item 49: How seriously appraiser took appraisal, 3 point scale. 

Significant at the .05 level 
Significant at the .01 level 
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^^^^^ — Relationship of Planning for Evaluation to Effectiveness of Principal Evaluation 



Measures of Effectiveness* 
No. of 



Planning for 


Item 


EFFPR 


50 


TEAC78 


DIR66 


Sig. r/ 


Evaluation 


b 

n 


14 


27 


26 


16 


Sign 




Mean 


2.21 


^ . ^0 


1 Q1 


5*25 






s.d. 


.58 


.42 


.52 


.68 




Method of Notification 










In person 


56.6% 
21 .0 


-.30 


-.27 


.39 


.37 


0 


By memo 


27.6% 
25.8 


.20 


-.05 


• 

.33 


* 

-.46 


2 ♦ 
1 - 


Visits 


%.f>% 
21 .2 


.18 


.05 


.02 


« 

.38 


1 ♦ 
1 - 


Statement 
on school 


58.6? 
25.4 


•31 


-.01 


.16 


.19 


2 - 


Pre-evaluation conference 










Conf. held 


68. 0? 
25.8 


.00 


.23 


.01 


-.10 


2 ♦ 


Length 


48.6 -.17 
15.3 ffiin. 


.04 


* 

.37 


.20 


6 ♦ 


Objectives-based 












Utilized 


73.2? 
30.3 


.00 


.15 


.17 


-.14 


1 ♦ 
1 . 



EFFPR: The effectiveness of principal appraisal as reported by directors on screening 

questionnaire, 3 point scale. 
Item 50: Extent of improvement in principal's performance reported by principals, 4 point 

scale. 

TEAC78: Extent of improvement in principal's performance as reported by teachers, 4 point 

scale (reversed from questionnaire version). 
DIR66: Extent of improvement in principal's performance as reported by directors, 4 point 

scale. 

b 

The n m^y vary among correlations due to missing data. 

Significant at the .05 level 
Significant at the .01 level 
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Table 67: Relationship of Purposes and Criteria for Evaluation to Intervening Effects of 
Principa l Evaluation 

Intervening Effects 



Purposes and Item 35 45 46 47 48 49 

Criteria Mean 2.89 2.79 2.76 2.99 2.56 2.61 

s.d. .35 .15 .16 .24 .30 .27 



Purposes Clearly Given 



Yes 



72.0!? 
22.0 



.64 



«« 



** «* 

.59 .64 .60 .58 



Criteria Used 

Admin. 88. 8? 

perforraance 25.8 

School/ 87.3? 

coiwDunity 12.7 
reins. 

Personnel 82. 3^ 

management 25 -4 



Contrib. 
to board 



78. 8!? 
16.7 



Contrib. 33.3 
to relig. 38.3 
education 



-.23 
.25 

.10 

.31 
-.1 1 



,12 



.22 



.27 



-.04 



.28 c42 



.23 
,16 
.22 



.05 
.29 

.27 
.06 



-.03 
-.36 

-.26 
-.24 



.18 -.07 



.09 
.17 

.09 
.06 
.22 



^ The n may vary among correlations due to missing data. 

^ Item 35: Principal satisfaction with report form, 4 point 
scale. 

Item 45: Fairness of appraiser's judgement, 3 point scale. 

Item 46: Fairness of procedures, 3 point scale. 

Item 47: Skilfulness of appraiser, 4 point scale. 

Item 48: How seriously principal took appraisal, 3 point scale. 

Item 49: How seriously appraiser took appraisal, 3 point scale. 

Significant at the .05 level 
Significant at the .01 level 
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Table 68: Relat i onship of Purposes and Criteria for Evaluation to Effectiveness of Principal 
Evaluation 



Measures of Effectiveness 

No. of 



Purposes and 


Item 


EFFPn 


50 


TEAC78 


DIH66 


Sig. 


Criteria 


b 

Q 


14 


27 


28 


16 


Sign 




Mean 


2.21 


2.28 


1 .91 


3.25 






s.d . 


.58 


.42 


.52 


.68 




Purposes Clearly Given 










Yes 


72. 


-.31 


-.05 


.15 


-.15 


6 ♦ 




22.0 












Criteria Used 














Admin. 


88.82 


.34 


• ?^ 
- . t> 


— . CO 


• 

4 C 




performance 


25.8 












School/ 


87.32 


.53 


-•23 


.17 


• 

.51 


2 ♦ 


comuni ty 


12.7 










1 - 


reins. 
























«« 




Personnel 


82.3% 


.40 


-.18 


.03 


.59 


1 ♦ 


management 


25.4 














78.82 


• 






«« 




Contrib. 


.57 


.05 


.15 


.68 


3 ♦ 


to board 


16.7 












Contrib. 




.26 


.26 


.03 


.26 


0 


to relig. 


38.3 













education 



EFFPR: The effectiveness of principal appraisal as reported by directors on screening 

questionnaire, 3 point scale. 
Item 50: Extent of improvement in principal's performance reported by principals, 4 point 

scale. 

TEAC78: Extent of improvement in principal's performance as reported by teachers, 4 point 

scale (reversed from questionnaire version) 
DIR66: Extent of improvement in principal's performance as reported by directors, 4 point 

scale. 

^ The n may vary among correlations due to missing data. 

Significant at the .05 level 
Significant at the .01 level 
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Most of the effects of pre-conferencing are confined to intervening variables. Both the 
presence of such a conference and its length are related to the principal's satisfaction with 
the report form and the fairness of procedures; the length of the conference is also related 
to the principal's views of the fairness of the appraiser's judgement, the aopraiser's skill, 
how seriously the appraiser took the process, and the extent of improvement in the principal's 
performance as a result of the appraisal as perceived by teachers. 

Results CG.^cerning purposes and criteria suggest two correlates of an effective appraisal 
system. Principals' reports that the purposes were clearly given correlate strongly with all 
intervening effects. Only one of the criterion domains used, the contribution of the 
principal to the board, appears related to variations in the effectiver^s of the process; it 
correlates positively with one intervening variable (the fairness of the appraiser's 
judgement) and two measures of effectiveness, the director's report of the effectiveness of 
the appraisal system as reported both on the screening questionnaire (EFFPR) and in the 
director's survey questionnaire (DIR66). The last variable, DIR66, is also positively 
corrc.uted with three other criterion domains: administrative performance, school and 
community activities, and personnel management. It would appear that directors view as 
effective those principal appraisal systems that are most comprehensive. Principals' reports 
of the effectiveness of their appraisal are not significantly correlated with any of the 
planning variables. 

2 Data Collection : The analysis of the possible effects of different sources and types 
of information used in principal appraisal is reported in tables 69 and 70; that for the 
effects of different collectors and the time spent on the evaluation process is in tables 71 
and 72. 
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Table 69: Rolation s hip of Sources and Types of Information Used to Intervening Effects of 
Pr i nc ipal Evaluation 



(n = 27T 



Intervening Effects^ 



Sources and 


Item 


35 


45 


46 


47 


48 


49 


Types of 


Mean 


2,89 


2.79 


2.76 


2.99 


2,56 


2,61 


Information 


s .d. 


.35 


.15 


.16 


.24 


.30 


.27 



Teachers Provide Information 



Yes 


23.92 
26.8 


9 


.09 


. 1 b 


* 

• 33 


- .04 


-.08 


ypes of Infonnation 














Checklist 


17.3?: 

17.2 


- .U 1 


. 15 


-.03 


- .22 


*« 

-.44 


-.23 


Specific 


61 .1% 


. 1 9 


.20 


• 15 


*« 

.47 


** 

.43 


** 

.45 


notes 


26.7 














General 


59.62 


.03 


.06 


.08 


* 

.39 


* 

.37 


* 

.35 


notes 


22.8 














Self-eval. 


19. 9S 


-.27 


-.05 


-.02 


-.14 


-.16 


-.28 


question. 


26.4 














Prev. app. 


15.0? 


.10 


* 

-.38 


-^15 


.02 


.09 


.08 


report 


18.6 














Disciplin- 


12.92 


-.17 


-.24 


-.30 


.06 


.45 


* 

.34 


ary records 


14.0 














School 


24.82 


.14 


-.02 


-.12 


.22' 


.25 


.25 


handbook 


22.9 












Goal 


70.42 


.17 


-.11 


.09 


.15 


-.28 


.01 


package 


23.0 















The n may vary among correlations due to missing data. 



scale. 

Item 45 

Item 46 

Item 47 

Item 48 

Item 49 



Principal satisfaction with report form, 4 point 

Fairness of appraiser's judgement, 3 point scale. 

Fairness of procedures, 3 point scale. 

Skilfulness of appraiser, 4 point scale. 

How seriously principal took appraisal, 3 point scale. 

How seriously appraiser took appraisal, 3 point scale. 



Significant at the .05 level 
Significant at the .01 level 
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Table 70: Relationship of Sources and Types of Information Used to Effectiveness of 
Principel Evaluation 



Measures of Effectiveness 



a 



No. of 



Planning for 


Item 


EFFPR 


50 


TEAC78 


DIR66 


Sig. 


Evaluation 


b 

n 


14 


27 


28 


16 


Sign 




Mean 


2.21 


2.28 


1 .91 


3.25 






s.d. 


.58 


.42 


• 52 


.68 




Teachers Provide Information 










Yea 


23.9? 


• 

.54 


-29 


.30 


.07 


3 ♦ 




26.8 












Types of Information 












Checklist 


17-3? 




.17 


.13 


.57 


1 ♦ 




17.2 










1 . 
















Specific 


61 .1% 


.31 


.42 


.08 


.40 


4 ♦ 


notes 


26.7 












General 


59. 6^ 


.29 


♦ 

• 38 


-.06 


.29 


4 ♦ 


notes 


22.3 






♦ 






Self-eval. 


19.9? 


.17 


-.17 


..36 


-.41 


1 - 


nuestion. 


26.4 












Prev. app. 


15.0? 


.29 


-.06 


.16 


.12 


1 - 


report 


18.6 






• 






Disciplin- 


12.9? 


.09 


.04 


..38 


-.09 


2 * 


ary records 


14.0 










1 - 


School 


24.8? 


.13 


-.19 


-.14 


.18 


0 


handbook 


22.9 












Goal 


70.4? 


.34 


-.04 


.28 


.16 


0 


package 


23.0 













^ EFFPR: The effectiveness of principal appraisal as reported by directors on screening 
questionnaire, 3 point scale. 
Item 50: Extent of improvement in principal's performance reported by principals, 4 point 
scale. 

TEAC78: Extent of improvement in principal's performance as reported by teachers 4 point 

scale (reversed from questionnaire version). 
DIR66: Extent of improvement in principal's performance as reported by directors, 4 point 

scale. 

^ The n may vary among correlations due to missing data. 

Significant at the .05 level 
Sionificant at the .01 level 
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One possible information provider is identified in tables 69 and 70, namely, teachers. 
The extent of their involvement in providing information is positively correlated with two 
intervening and one final-effect variable, providing some evidence that asking teachers about 
their principal is a useful component of a principal appraisal system. 

General and specific notes taken by the appraisers are the only two types of information 
that seem consistently related to the effects of the appraisal process. Both are positively 
correlated with principals' own perceptions of the extent of their improvement as a result of 
being appraised. In addition, the utilization of student disciplinary records as part of the 
information base for the appraisal is positively related to how seriously the process is 
perceived by principals. However, this variable is negatively correlated with the improvement 
in a principal's performance as perceived by teachers. 

The use of one of the most popular types of information, goal packages, is not positively 
correlated with any of the effectiveness variables; neither is the use of school handbooks. 
Results for checklists (one positive and one negative) are mixed, though it is perhaps notable 
that principals in boards using such instruments appear to take evaluation less seriously than 
do others, while directors in such boards are likely to view their principal appraisal systems 
as being more effective. As well, use of two types of information that were used less in 
practice than either principals or directors would prefer self-evaluation and use of the 
previous appraisal report each have one negative correlation, suggesting little or no 
pjcial impact from their use. 
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Table 71: Relationship of Who Collected Information and Time Spent on Evaluation to 
Intervening Effects of Principal Evaluation 
(n = 27)^ 



Intervening Effects 



Who Collected 



Inf onsa t ion 










47 


46 


49 


and 


Mean 


2.69 


2.79 


2.76 


2.99 


2.56 


2.61 


Time Spent 


s.d. 






.16 


.24 


.50 


.27 


Team Used 
















Yes 


21 .0 


.25 


.00 


-.03 


« 

.34 


.21 


.20 


Individuals 


Highly 


[nvolved 












Teachers 


10.4;? 
17.4 


.19 


« 

.40 


.66 


«« 

.59 


.12 


.21 


Area 
Supt* 


85.22 
26.1 


.22 


.10 


.09 


.16 


.07 


.00 


Other 


1.6% 


.08 


.06 


.03 


.29 


.20 


.14 



supt. 20.7 
Number of Days Spent Collecting Information 

.21 .21 



Ave. no. 
of days 



3.1 .12 
1 . 1 days 



Post-evaluation Conference Length 



Length 



54.5 .49 
23.5 min. 



.11 



.00 



.29 



.15 



.44 



.39 



.32 



.46 



^ The n may vary among correlations due to missing data. 

^ Item 35: Principal satisfaction with report form, 4 point 
scale. 

Item 45: Fairness of appraiser's judgement, 3 point scale. 

Item 46: Fairness of procedures, 3 point scale. 

Item 47: Skilfulness of appraiser, 4 point scale. 

Item 48: How seriously principal took appraisal, 3 point scale. 

Item 49: How seriously appraiser took appraisal, 3 point scale. 



Significant at the .05 level 
Significant at the .01 level 
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l^^^^ Relationship of Who Collected Information and Time Spent o n Evaluation to 

Lffectiveness of Principal Evaluation " 



Who Collected 



Measures of Effectiveness* 



No. of 



Information 


Item 


EFFPR 


50 


TEAC78 


DIR66 


Sig. r/ 


and 




14 


27 


28 


16 


Sign 


Time Spent 


Mean 


2.21 


2.28 


1.91 


3.25 






s.d. 


.58 


.42 


.52 


.68 




Tean Used 














Yes 


21 .0 


.38 


-.09 


.08 


• 


p ♦ 
c ^ 


Individuals 


Highly Involved 










Teachers 


10.4!? 
17.4 


.30 


.21 


.22 


- .23 


3 ♦ 


Area 
Supt. 


85.22 
26.1 


.39 


-.01 


.23 


-.13 


0 


Other 
supt. 


7.6$ 
20.7 


« 

.50 


-.13 


-.06 


.06 


1 ♦ 


Nunber of Days Spent 


Collecting Information 






Ave. no. 
of days 


3.1 .21 
1 . 1 days 


.22 


.07 


>19 


2 ♦ 


Post-evaluation Conference Length 








Length 


54.5 -,38 
23.5 min. 


• 

.36 


.18 


.02 


4 ♦ 



EFFPR: The effectiveness of principal appraisal as reported by directors on screening 

questionnaire, 3 point scale. 
Item 50: Extent of improvement in principal's performance reported by principals, 4 point 

scale. 

TEAC78: Extent of improvement in principal's performance as reported by teachers, 4 point 

scale (reversed from questionnaire version). 
DIR66: Extent of improvement in principal's performance as reported by directors, 4 point 

scale. 

The n may vary among correlations due to missing data. 

*A Significant at the .05 level 
Significant at the .01 level 
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Tables 71 and 72 reveal more positive effects from characteristics related to the 
identity of those collecting information and the time they spend doing so. Use of an 
appraisal team is correlated with the seriousness with which principals take the process and 
directors' ratings of the amount of improvement that comes about in a principal's performance 
as a result of appraisal, fs well, more involvement of teachers in the entire process (not 
necessarily just as collectors of information) is positively related to the fairness of the 
process and the appraiser's judgement, and to the skill with which the evaluation is carried 



out. 



Time, again, proves an important characteristic. The number of days spent collecting 
information is positively related to how seriously principals take the process and how 
seriously th3y perceive it is taken by their appraisers. The length of the post-evaluation 
conference, if one is held, appears still more important; it is positively related to 
principals' satisfaction with the appraisal form, seriousness of the process, and principals' 
ratings of their own improvement. 

3. Reporting and Fol low-Up : Tables 73 and 74 describe the relationship of reporting and 
follow-up activities with the perceived effectiveness of various components of principal 
appraisal schemes. 
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^^^^^ Re lationship of Reporting and FoUow-Up to Interven ing Effects of Principal 

Evaluation — 

(n = 27)^ 



Intervening Effects^ 



Reporting 


Item 


35 


45 


46 


47 


48 


49 


and 


Mean 


2.69 


2.79 


2.76 


2.99 


2.56 


2.61 


Follow-up 


s.d. 


.35 


.15 


.16 


.24 


.30 


.27 


Post-evaluati 


on Conference 












Yes 


15-9 


. 1 1 


-.19 


.00 


.04 


.03 


.12 


Report Type 
















Several 
headings 


39-2? 
25.7 


- .01 


-.13 


-.10 


.15 


.15 


.19 


Hn <? t rijc— 

UIXO Itf 1 u\» — 

tured 


16.3 


.08 


• 

.35 


. 1 8 


-02 


-.15 


.01 


Rating of 
activities 


6.32 
16.0 


-.08 


-.01 


-.09 


-.23 


-.40* 


-.45* 


Summary 
mark 


2.12 
6.9 


.01 


.03 


-.03 


-.01 


-.28 


".25 


No report 


32.52 
33.6 


-.01 


-.03 


-.02 


.05 


.16 


.08 


Appeal Process 














Yes 


30.82 
25.8 


.07 


•25 


.50 


.10 


-.12 


-.24 


Follow-up 
















Plan 

developed 


29.1 
20.9 


«« 

.54 


.16 




.51 


.17 


.30 


Plan 

monitored 


64.02 
28.0 


.05 


.01 


-.02 


.01 


.08 


.08 


Prof. dev. 


67.82 
31 .6 


-.19 


-.12 


.07 


.04 


.28 


.30 


Letter of 
commend • 


10.6 
15.9 


.11 


.05 


-.04 


.09 


-.12 


.13 



^ The n may vary among correlations due to missing data. 

^ Item 35: Principal satisfaction with report form, 4 point 
scale. 

Item 45: Fairness of appraiser's judgement, 3 point scale. 

Item 46: Fairness of procedures, 3 point scale. 

Item 47: Skilfulness of appraiser, 4 point scale. 

Item 48: How seriously principal took appraisal, 3 point scale. 

Item 49: How seriously appraiser took appraisal, 3 point scale. 

Significant at the .05 level 
Significant at the .01 level 
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Table 74: Relationship of Reporting and FoUow-Up to Effectiveness of Principal Evaluation 



Keaaurea of Effectiveness 

No. of 



Reporting 


Item 


EFFPR 


50 


TEAC78 


DIR66 


Sig. r/ 


and 


b 

n 


14 


27 


28 


1 6 


Sign 


Follow-up 


Kean 


2.21 


2 . 23 


1 .91 


3.25 






s.d. 


.58 


. 42 


.52 


.68 




Post-evaluation Conference 














** 


« 




-.17 




Yea 


84.6$ 


.56 


.41 


-.05 






15.9 












Report Type 














Several 


39.22 


.44 


-.23 


-.37 


. 1 6 


1 - 


headings 


25.7 












Unstruc- 


18.32 


-.20 


-.04 


.00 


. 1 8 


1. ♦ 


tured 


16.3 












Rating of 




. 26 


-.33 


-.15 


.32 


2 - 


ac tivi ties 


16.0 












Summary 


2.12 




-.27 


-.29 


.27 


0 


mark 


6.9 






« 






No report 


32.52 


1 1 
- . 1 / 


.30 


.33 


-.15 


1 ♦ 


document 


33-6 












Appeal Process 












Yes 


30.82 


.37 


-.29 


.25 


.21 


U 




25-8 












Follnw-up 




















« 


« 






plan 


29-1^ 


.36 


.61 


.36 


-.17 


4 ♦ 


developed 


20.9 












Plan 


64.02 


-.16 


-.29 


.28 


.03 


0 


laoni tored 


28.0 












Prof. dev. 


67.82 


-.26 


.27 


.28 


-.15 


0 




31.6 












Letter of 


16.62 


-.11 


-.20 


.12 


.15 


0 



commend . 1 5 -9 



^ EFFPR: The effectiveness of principal appraisal as reported by directors on screening 
questionnaire, 3 point scale. • • i zi r.r.\r.^ 

Item 50: Extent of improvement in principal's performance reported by principals, 4 point 

TEAC78: Extent of improvement in principal's performance as reported by teachers, 4 point 
scale (reversed from questionnaire version). /, • ♦ 

DIR66: Extent of improvement in principal's performance as reported by directors, 4 point 
scale. 

^ The n may vary among correlations due to missing data. 

\^ Significant at the .05 level 
Siqnificant at the .01 level 
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At the end of a principars appraisal, a post-evaluation conference was typically held 
between the principal and the appraisers. The existence of such a conference is positively 
related to two of the measures of effectiveness of appraisal systems: the director's rating 
of the effectiveness of the system as indicated on the screening questionnaire and principals' 
own self-reports of their improvement as a result of the appraisal. As well, we have already 
seen that the length of such conferences is important. 

The particular form of statement given the principal does not seem to matter: statements 
under several headings, unstructured statements, and no form at all each display one 
significant correlation with either intervening or end-result measures of effectiveness. 
Forms with scale ratings of various activities, however, are negatively correlated with how 
seriously the evaluation process was looked upon by principals. 

The existence of an ap^Jeal process s not correlated with any of the variables measuring 
the effectiveness of the evaluation process, but development of a plan for the improvement of 
the principal's performance has four significant positive correlations. Among the correlates 
are satisfaction expressed regarding the report, fairness of the procedures, and amount of 
improvement made by the principal as perceived both by the principals themselves and their 
teachers. Whether or not a plan was monitored or specific professional or academic 
activities were undertaken to assist in implementing the plan were not significant factors; 
neither was the receipt by the principal of a letter of commendation. 

4. Evolution of Policy : Tables 75 and 76 report findings concerning the relationship of 
the approach taken in developing and implementing an appraisal system to its effectiveness. 
The first group of items concerns who participated in the development of the policy. 




Table 75: Relationship of Evolution and Implementation of Policy to Intervening Effects of 



Principal Evaluation 
(n = 27)^ 



Intervening Effects 

Evolution and 



Implement- 


Item 


35 


45 


46 


47 


48 


49 


ation of 


Mean 


2.89 


2.79 


2.76 




2 . S6 


2.61 


Pol ^cy 


s.d. 


.35 


• 15 


. 1 0 


.24 


.30 


.27 


Participants 


in Policy Development 


















« 


«« 


« 


« 


Trus t ees 


18. 2? 


-.16 


.31 


• JO 


A A 

. 44 


. :5o 






14.9 














jirec tor 


73-4^ 


.03 


-.14 


.0 I 


. ^5 




. 




19.6 














Supt's. 












«« 






.05 


.19 


.10 


-.23 


-.56 


-.52 




26.7 














Principals 


61 .^% 


.24 


.23 




« 


- . Ub 


.U 1 




54-5 


















« 












Teachers 


H.2% 


-.37 


-.10 


- . ub 






. 1 2 




15.0 














Efforts to Implement 


Policy 












Supporting 


57.5^ 


-.23 


.00 


« 

-.05 


.04 


-.09 


-•13 


documents 


37.1 
















57.1? 






«« 








Workshops 


-.14 


.25 








-.11 




25.9 














Resources 


39-0? 


-.08 


.16 


« 

.39 


« 

.37 


.12 


•25 




28.8 














Fidelity of 


Impelementation 






















»« 






As m 


25. 9? 


.32 


.27 


.37 


.44 


.24 


.29 



policy 30.3 



^ The n may vary among correlations due to missing data. 

° Item 35: Principal satisfaction with report form, 4 point 
scale. 

Item 45: Fairness of appraiser's judgement, 3 point scale. 

Item 46: Fairness of procedures, 3 point scale. 

Item 47: Skilfulness of appraiser, 4 point scale. 

Item 48: How seriously principal took appraisal, 3 point scale. 

Item 49: How seriously appraiser took appraisal, 3 point scale. 

Significant at the .05 level 
Significant at the .01 level 
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Table 76: Re l ationship of Evolution and Implementation of Policy to Effect iveness of 
Principal Evaluation 



Evolution and 



Measures of Effectiveness^ 



No. of 



Implement- 


Item 


EFFPR 


50 


TEAC78 


DIR66 


Sig. r/ 


ation of 


b 

n 


14 


27 


28 


16 


Sign 


Po 1 1 cy 


Mean 


2.21 


2.28 


1 .91 


3.25 






s.d. 


.58 


.42 


.52 


.68 




Participants 


in Policy Development 








Trustees 


18.2? 
14.9 


-.56 


-.13 


.22 


-.08 


4 + 
1 . 


Director 


73.4? 
19.6 


-.18 


* 

.32 


-.15 


.22 


1 + 


Supt*s. 


85.2? 
26.7 


.13 


-.15 


.01 


.18 


2 - 


Principals 


61.1? 
34.5 


.07 


- 1 fs 


-.04 


.07 


1 + 


Teachers 


14.2? 
15.0 


.06 


.00 


.42 


.21 


1 + 
1 « 


Efforts to Implement 


Policy 










Supporting 
documents 


57.5? 
37.1 


-.27 


-.28 


.56 


.20 


1 + 


Workshops 


57.1? 
25.9 


-.09 


«13 


.01 


.02 


1 + 


Resources 


39.0? 
28.8 


-.05 


.08 


.16 


.07 


2 + 


Fidelity of Impelementa tion 










Aa in 


25.9? 


** 

.68 


.13 


* 

.07 


-.24 


3 + 



policy 30.3 



EFFPR: The effectiveness of principal appraisal as reported by 

directors on screening questionnaire, 3 point scale. 
Item 50: Extent of improvement in principal's performance 

reported by principals, 4 point scale. 
TEAC78: Extent of improvement in principal's performdnce as 

reported by teachers, 4 point scale (reversed from 

questionnaire version). 
DIR66: Extent of improvement in principal's performance as 

reported by directors, 4 point scale. 

b -r.. 

The n may vary among correlations due to missing dd^j. 

Significant at the .05 level 
SignitiLat.L at the .01 level 
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Involvement of trustees stands out as one group whose participation, though rare (18.2 
per '"ent on average imong the boards), is positively correlated to the intervening affects of 
evaluation (fairness and seriousness), though negatively correlated with one end-result 
measure. Invol vemera of superintendents, however, is negatively rplateH how seriously the 
appraisal process is taken. 

Efforts made to implement the appraisal policy, including documentation, workshops, and 
devotion of considerable rescjrces, each have one or two positive correlations with the 
measures of impact. Fidelity of implementation appears more important", having three 
significant correlations. 

Overall, the relationship of the steps taken to develop and i.nplement a policy with the 
policy's effectiveness is not particularly strong; nevertheless, the data do suggest that a 
strong commitrr.ent , as expressed by the involvement of and resources from those at the top, is 
important. 

5. Demographic Variables : Numerous backgroun 1 varia*^les are correlated with the measures 
of effectiveness for principal evaluation systems. A few variables suggest a pattern of 
responses that implies older, experienced male principals, especially in secondary schools, 
taKe the process rather less seriously than do others; however, no variable accounts for much 
variation in the effectiveness of the final results of the process. 

The first group of variables reported in tables 77 and 78 involves school system 
characteristics. There is never more than one significant correlation for any one variable; 
such evidence suggests school type, enrolment, year in which the policy was adopted, rate of 
enrolment decline, and percentage of French-language schools in a board are not relevant to 
the effectiveness of principal e\.aluation. 
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Table 77 : Relationship of Background Variables to Intervening Effec ts of Principal Evaluation 
(n = 27)^ 



Intervening Effects 



Background Item 


35 


45 


46 


47 


48 


41 


Variables Mean 


2.89 


2.79 


2.76 


2.99 


2.56 


2.61 


3-d. 


.35 


.15 


.16 


.24 


.30 


.27 


School System Characteristics 












Type 46.7? 
(0 « pub; 50.'/ 
1 ■ sep.) 


-.02 


.04 


.24 


.09 


-.05 


• 

-.05 


1982 20,922. 
enrol. 24,906. 


.03 


.31 


.15 


.09 


-.1^ 


-.10 


Year 1978. 
adopt 3.5 


.10 


.08 


.03 


.01 


.00 


-.06 


Rate enrol. 2.22 
decline 34.5 


-.03 


.07 


.00 


• 

-.j4 


-.16 


-.26 


Per :entage 9.4S 


-.18 


..09 


-.12 


-.20 


-.04 


-.06 



Fr. schools 16.5 
Principa 1 Characteristics 

-.06 



Average 
age (10 
pt. scale) 

Percentage 
w/ degree 

Percentage 
w/ spec . 

Mean yrs. 

ia school 

Mean yrs. 

in board 

Mean yrs. 
as a prin. 

Mean yrs. 
in educ. 

Percentage 
males 

Percentage 
pern. appt. 



.32 
.06 
.04 



5.4 
.6 

95. 9^ 
9.6 

71 At 
14.3 

6.1 
3.9 years 

18.4 -.17 
3.2 years 

12.2 .01 
2.7 year^ 

22.6 -.01 
2.7 years 

32 



86.0*^ 



15.3 

93. 9? 
6.7 



.41 



.27 

".13 
".37 
.02 
«14 
-.08 
.22 

« 

-.45 
.14 



• 

.06 -.10 -.37 -.44 



-.19 
-.41 
-.01 

.06 
-.14 

.05 

« 

-.57 
.17 



.44 -.46 -.40 



.13 
.10 
.18 
.18 
.07 

• 

.47 



.01 
.49 
-.40* 
-.49 
-.33 
-.19 



.19 -.16 



.14 
.42 

« 

-.50 

i 

-.51 

« 

-.36 
-.20 
-.24 
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Table 77 (continued) 



Interveamg Effects 



Background 


Item 


35 


45 


46 


47 


48 


41 


Variables 


Mean 


2.89 


2.79 


2.76 


2.99 


? .00 


^ .b 1 




s.d. 


.35 


.15 


.16 


.24 




.27 


Percentage 


15.1? 


.19 


.22 


.22 


.26 


.26 


« 

.37 


time teach. 


14.9 














School 


392.6 


-.12 


.10 


-.05 


-.10 


-.19 


-.18 


enrolment 


124.5 
























« 




-.29 


Highest 


8.0 


-.31 


-.41 


-.47 


-.40 


-.29 



gr8de .8 



^ The n may vary among correlations due to missing data. 

^ Item 35: Principal satisfaction with report form, 4 point 
scale. 

Item 45: Fairness of appraiser's judgement, 3 point scale. 

Item 46: Fairness of procedures, 3 point scale. 

Item 47: Skilfulness of appraiser, 4 point scale. 

Item 48: How seriously principal took appraisal, 3 point scale. 

Item 49: How seriously appraiser took appraisal, 3 point scale. 

Significant at the .05 level 
Significant at the .01 level 



Table 78: Relationship of Background Variables to Effectiveness of Principal Evaluation 



Measures of Effectiveness 



No. of 



Variables Item 


EFFPR 


50 


TEAC78 


DIR66 


Sig. r/ 


b 

n 


14 


27 


28 


16 


Sign 


Mean 


2.21 


2.28 


1 .91 


3.25 




s.d. 


.58 


.42 


.52 


.68 




School System Characteristics 










Type 46.75? 
(0 « pub; 50.7 
1 - sep.) 


-04 


.14 

« 


• 

.32 


-.10 


1 ♦ 


1982 20.922. 


-.07 


-.35 


-.20 


.06 


1 - 



enrol. 24,906. 
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Table 78 (continued) 

Background 



Measures of Effectiveness 



No. of 



variaDles 


Item 


SFFPR 


50 


TEAC78 


DIR66 


Sig. r/ 






14 


27 


28 


16 


Sign 




Mean 


2.21 


2.28 


1 .91 


3.25 






s.d . 


.58 


.42 


.52 


.68 




Year 

adopt 


1 978. 
3.5 


.04 


- .22 


* 

- . 


— .11 


1 - 


Rate enrol 
decline 


. 2.2% 
34.5 


.09 


.08 


-.28 


-.06 


0 


Percentage 9.4!? 
Fr. schools 16.5 


. bU 


-.35 


-.24 


•19 


1 ♦ 


Principal Characteri 


sties 










Average 
age (10 
pt* scale) 


5.4 
.6 


.24 


« 

-.35 


-.29 


.28 


3 - 


Percentage 
w/ degree 


95.95 
9.6 


•19 


.07 


.00 


-.16 




Percentage 
w/ spec. 


71 .4^ 
14.3 


.00 


.02 


« 

-.22 


«« 

.70 


\ ^ 

2 - 


Mean yrs. 
in school 


6.1 -.12 
3.9 years 


.09 


.02 


-.31 


c ^ 


Mean yrs. 
in board 


16.4 .18 
3*2 years 


« 

-.39 


« 

- .42 


.01 


k _ 
•f - 


Mean yrs. 
as a prin. 


12.2 -.25 
2.7 years 


-.14 


-.17 
— .If 


DA 


c - 


Mean yrs*. 
in educ. 


22.6 .04 
2.7 yea s 


« 


. 




3 - 


Percentage 
males 


86. 0^ 
15.3 


-.15 


.03 


-.22 




J 


Percentage 
perm. appt. 


93. 9^ 
6.7 


.24 


-.17 


.13 


-.01 




Percentage 
time teach. 


15. 1? 
14.9 


-.18 


.22 


.06 


-.41 


1 ♦ 


School 
enrolment 


392.6 
124.5 


.41 


• 

- .^0 


— . \Ju 




4 

1 - 


Highest 
grade 


8.0 
.8 


.06 


.08 


-.20 


.08 


5 - 



EFFPR: The effectiveness of principal appraisal as reported by directors on screening 

questionnaire, 3 point scale. 
Item 50: Extent of improvement in principal's performance reported by principals, 4 point 

scale. 

TEAC78: Extent of improvement in principal's perfo nee as reported by teachers, 4 point 

scale (reversed from questionnaire version). 
DIR66: Extent of improvement in principal's performance reported by directors, 4 point 

scale. 

b ^ 

The n may vary among correlations due to missing data. 

n 

Significant at the .05 level 
Significant at the .01 level 
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Principal (and school) characteristics are somewhat more telling. The average age of and 
percentage of degree holders among a school board's principals are negatively correlated with 
how seriously appraisal is taken. Average age is negatively correlated with the improv3ment 
reported as a result of evaluation. Boards with principals who have been in their school a 
longer period than average, however, have principals who report taking the process more 
seriously. Other variables measuring experience have negative relationships, including 
average number of years with the board, average number of years as a principal, and years of 
experience as an educator. Percentage of males and highest grade taught also have negative 
relationships. Variables with only one significant correlation include percentage of 
principals with permanent contracts, percentage of time the principal spends teaching, and the 
average size of a school in a board; these three, then, are not important in explaining 
effectiveness. 

Overall, the results of the demographic analysis suggest a pattern also found among 
teachers, though the pattern seems weaker in the case of principals. That pattern is one in 
which men, particularly more experienced men with degrees at the secondary level, find the 
evaluation process to be less helpful than do others. 

Performance Appraisal of Superintendents 

A sample of 214 supe.i ntendents in 28 Ontario boards — 15 public and 13 Roman Catholic 
separate — was asked to reply to the superintendent questionnaire; two boards in the overall 
sample of 30 did not have superintendents. Completed questionnaires were received from 114 
superintendents in 25 boards - 13 public and separate - thereby yielding an overall 
return rate of 53.3 per cent. Regions represented by the responding superintendents are shown 
in table 79. 

Table 79: Distribution of Superintendents by Region and by Type of Board 



KLC 



Region 


Public 
(n«79) 
n % 


Separate 

(n-35) 
n % 


Total 
(n-IU) 
n % 


Northern 


4 


5.1 


5 


14.3 


9 


7.9 


Western 


9 


11 .4 


2 


5.7 


1 1 


9.6 


Central 


49 


62.0 


25 


71.4 


74 


64.9 


Eastern 


17 


21.5 


3 


8.6 


20 


17.5 


Total 


79 


69.3 


35 


30.7 


114 

223 


100.0 



Seventy-four per cent of the superintendents indicated that their board conducted formal 
performance appraisals of superintendents. Appraisals were more common for superintendents in 
public boards (80 per cent) than for those in separate boards (60 per cent). Twelve of the 13 
public boards conducted superintendent appraisals while only 5 of the 12 separate school 
boards did so. Overall, then, 68 per cent of the boards carried out this practice. 

For the most part, then, the data reported here relate to those 17 boards carrying out 
superintendent appraisals. Overall, 72 of the 114 superintendents, or 63 per cent, had been 
appraised recently. Of these, 57 (79 per cent) were in public boards and 15 (21 per cent) 
were in separate boards. 



Preparation for appraisal includes planning, pi-rposes, criteria, and standards as 
subcategories. 



In the majority of cases (67 per cent), superintendents were notified of a coming evaluation 
in a request for a statement about their objectives, activities, and plans. Notification in 
person (35 per cent) or by memorandum (35 per cent) was not unusual, but a request for 
self-evaluation (18 per cent) or informal visits (19 per cent) was. 

Only 35 per cent of the superintendents reported pre-conferences were held between them 
and their appraiser(s) before the process began; there was little difference between public 
and separate boards. When conferences were held, they averaged 49 minutes in length ( s.d. = 
26 minutes) and tended to be longer in separate boards (64 minutes) than in public boards (45 
minutes). 

Sixty-three per cent of the superintendents (74 per cent in separate and 60 per cent in 
public boards) who had been appraised in recent years reported that the setting of objectives 
had been a central part of their last evaluation. Of these, 27 per cent reported the 



Preparation 



Planning 
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objectives focussed on them personally, 14 per cent on their unit as a whole, and 59 per cent 
on both. The last response was more common in separate boards (73 per cent) than public 
boards (55 per cent). 



Two-thirds of the superintendents reported that the purposes of their last appraisal were 
clearly communicated to them beforehand. These purposes were to: assess the achievement of 
their objectives, 81 per cent; comply with board policy, 63 per cent; identify administrative 
weaknesses in need of improvement, 56 per cent; assess achievement of their board's 
objectives, 51 per cent; identify administrative strengths, 47 per cent; reassure and 
develop self-confidence, 36 per cent; develop school/community relations, 35 per cent; 
improve student learning, 28 per cent; clarify superintendents' role, 26 per cent; identify 
in-service training needs, 25 per cent; and assess effectiveness of instructional program, 24 
per cent. The remaining choices in the questionnaire item were checked by 6 per cent or fewer 
of the respondents: to qualify superintendent? for regular increment, to select 
superintendents for promotion, to recommend renewal of contract, to identify superintendents 
for transfer, to establish evidence for demotion due to inadequate administrative performance, 
and to qualify for merit pay. 

In effect, the appraisal of superintendents was used exclusively for developmental 
purposes. 



The criteria that superintendents indicated were used and which they believed ought always be 
used are reported in table 80. 

Table 80: Actual and Ideal Criteria for Superintendent Appraisal, as R eported by 
Superintendents 



Purposes 



Criteria 



Criterion for Appraisal 



Actual 



(n-72) 



Ideal 
(n-114) 



General domains: 



Administrative performance 
School and community relations 
Program organization 
Personnel management 



81.92 



52.8 



93.3? 
65.7 



47.2 



68.8 



76.4 



87.4 
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Table 80 (continued) 



Criterion for Appraisal Actual Ideal 

(n»72) (n-114) 



Contribution to board 65.3 74.8 

Religious education 12.5 23.9 

Routine administration of; 

Program 56.9 '^B'T 

Budget 52. n 58.7 

Records 44.4 48.5 

Facilities 30.6 39*8 

Office staff 36.1 43.9 

Interpersonal relations with; 

Parents 36.1 44.7 

Teachers 43.1 60.4 

Principals 72.2 84.4 

Other superintendents 68.1 73*0 

Trustees 70.8 61 .8 

Director 68.1 32.9 

Federation and union officials 29.8 34.0 

Knowledge of; 

Board policies 48.6 74.8 

Admin' strative procedures 52.8 73*2 

Relevant acts and regulations 36.1 64.2 

Personnel 52.8 65. 1 

Community 41 .7 46.7 

Curriculum and program 45.8 67.6 

Schools i.i area of responsibility 63.9 92.5 

Skills: 

Management 69.4 90.1 

Supervisory 76.4 92.9 

Communication 72.2 91*9 

Organizing 66.7 87.3 

Decision-making 59*7 91*8 

Problem-solving 62.5 86.5 

Human relations 66.7 9O.O 

Program evaluation 41.7 70.6 

Personnel evaluation 63*9 77.3 
Other; 

Achievement of objectives 81 .9 

"personality 38.9 55.8 
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Overall, there is a fair parallel between actual and ideal factors, though the 
percentages for the latter tend to exceed those for the former quite consistently. With 
regard to knowledge of board policies and of relevant acts and regulations this difference is 
considerable, suggesting that these areas may currently receive too little emphasis. Only two 
criteria appear, in the superintendents' view*;, to be overemphasized: relations with trustees 
and the superintendent's personality. 

On only one item was there a large difference between public and separate 
superintendents. Four per cent of the former reported that their contribution to rel rgious 
education was a criterion used in their last appraisal; none thought it ought to be. In 
contrast, 47 per cent of the separate school superintendents reported this criterion had been 
used; 69 per cent thought it ought always to be. 

Standards 

Standards for performance on the various criteria were most often set by the appraiser (44 per 
cent), although 22 per cent (26 per cent in public and 7 per cent in separate boards) did not 
know how they were set. As well, 29 per cent reported that they were set collaboratively, 3 
per cent by the appraisee (none in the public and 13 per cent in the separate boards), and 1 
per cent by board guidelines. 

Data Col lection 

The data collection phase of performance appraisal is concerned with the sources of 
information, what types of information are collected, who collects th3 information, and the 
time spent in the process. 

Sources of Information 

Numerous groups were involved in providing information for the appraisal of some 
superintendents, though two individuals the superintendent and the director were most 
1 ikely to do so, 
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Complete responses to the questions of who does and who ought to provide information are 
reported in table 81. 

Table 81: Actual and Ideal Sources of Information for Superintendent Appraisal, as Reporte d 
by Superintendents 



Source 


Actual 
(n-72) 


Ideal 
(n-1 14) 


The superintendent being appraised 


86. Ii 


87. 7^ 


Director of education 


55-6 


87.7 


Trustees 


26.4 


51 .8 


Other superintendents 


15-3 


. 54.4 


Principals 


9.7 


64.0 


Teachers 


6.9 


26.3 


Board staff 


5.6 


28.9 


Parents 


4.2 


5.1 


Students 


1.4 


3.8 


Other 


4.2 


7.0 



Trustees were more likely to provide information in public boards (32 per cent) than in 
separate boards (7 per cent). This was the only difference between these boards that was of 
consequence. 

Overall, it appears that superintendents believe too few sources of information are 
currently used in their appraisal and that the number of sources should be increased, 
particularly among professionals within the board. 

Types of Information 

The actual and ideal types of inforration superintendents report being used in their 
performance appraisals are given in table 82. 
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Table 82: Actual and Ideal Types of Information Used in Superintendent Appraisal, as Reported 
by Superintendents 



Type of Inforaation Actual Ideal 

(n«72) (n=1 14) 

Objectives written by superintendent 84 "7? 85 -S? 

Objectives written by director 27.8 56.5 

Objectives written by board 19*4 49*5 

Interview by committee 0«0 6.7 

Questionnaire completed by principals 
or other educational staff that the 

superintendent supervises 1 .4 22.2 

Self-evaluation questionnaire 2.8 44.1 

Written self-evaluation 16.7 41.4 

Daily calendar or diary 13.9 11«1 

Standardized test scores for students 0.0 0.0 

Oral report 37.5 40.4 
Reports for which the superintendent 

had been responsible 38.9 52.3 

Written report from previous appraisal 13.9 41.6 

Reports on achievement of objectives 44.4 80.6 

Results of external assessment of board 0.0 9^^ 
Goal package the superintendent had 

prepared for self and unit 31 -9 70.9 



While superintendents, as has been noted, support the provision of information by others 
in their appraisal, the specific choices given on the questionnaire apparently did not reflect 
the types they w->uld prefer. For example, a relatively low percentage support the use of 
questionnaires completed by subordinates. At the same time, the data suggest that relatively 
little information of a formal sort is being used in the typical appraisal; only objectives 
written by the superintendent were used in a majority of the cases. 

Preferences also indicate superintendents would appreciate more direction and per\ aps 
role clarification from their director and school board. Further, they wished to be judged 
on results, particularly results as reflected in reports +hat are prepared by themselves in 
the course of their jobs or that address the achievement of their written objectives. 



212 



Collectors of Information 



The director is the sole collector of information and the appraiser in virtually all 
appraisals. Teams are never used. Others may be involved somewhere in the process, though: 
44 per cent of the superintendents indicated other superintendents were involved in their last 
appraisal; 43 per cent reported trustees; 18 per cent principals; and 4 per cent teachers. 

Time Spent 

Sixty-three per cent of the superintendents reported less than one day had been spent in 
collecting information for their last appraisal; yet, 17 per cent indicated more than five 
days were spent. Responses for the remaining 20 per cent were distributed between these 
extremes, with 10 per cent at one day and 6 per cent at five days. To this item, 25 per cent 
did not reply, an unusually high non response rate. Apparently, the process of evaluating a 
superintendent by a director is very different from that of appraising teachers and 
principals; it appears to be a process worked into the daily routine rather than an event 
that takes place at one point in time. This conclusion was confirmed in the case studies. 

Reporting and Follow-Up 

The nature of the reporting, including any form completed and filed, the sharing of this 
document, and the steps taken after the completion ^f a performance appraisal, are the topics 
of this section. 

Nature of Report 

Post-conferences, reported by 76 per cent of the superintendents who had had their 
performances appraised, were more common than were pre-conf erences . In the majority of cases 
(54 per cent), the director was present at the post-conference. In public boards, it was 
common to have someone else present as well (44 per cent). The questionnaire did not ask who 
this other person, or persons, might be, but in case studies it was apparent that in larger 
boards the associate director or senior academic superintendent ofttn played a part in the 



appraisal of superintendents. On average, the conferences lasted 49 minutes (s^ - 27 
minutes), though they were longer in separate boards (71 minutes) than in public boards (44 
minutes). 



Assessment of the quality of the post-conference was mixed: 36 per cent reported the 
conferences were very well planned, 46 per cent fairly well planned, and 18 per cent poorly 
planned. Free participation h> all parties was reported by 86 per cent of the evaluatees; 8 
per cent reported both were restrained; 4 per cent that the appraiser was involved and the 
appraisee was restrained; and 2 per cent that the appraisee was involved and the appraiser 
restrained. The experience was not at all threatening for 88 per cent of the superintendents 
while 12 per cent found it somewhat threatening. 

Seventy-eight per cent reported sincere praise had been given, 16 per cent found the 
praise insincere, and no praise was reported by 6 per cent. Constructive criticism was 
provided in 55 per cent of the cases, no criticism in 24 per cent, partly constructive in 14 
per cent, and totally unconstructi ve in 8 per cent. 

After the post-conference, 69 per cent of the superintendents felt good while 26 per cent 
felt neither good nor bad. Very few felt somewhat negatively (2 per cent) or very uneasy and 
defensive (4 per cent). All of the latter were in public boards. 

Written reports were provided to 64 percent of the superintendents; the practice was more 
comiT..n in public boards (72 per cent) than in separate boards (33 per cent). Twenty-five per 
cent were very satisfied with the type of the report used and 57 per cent were satisfied. 
Relatively few were dissatisfied (11 per cent) or very dissatisfied (7 per cent). 

Had they been dissatisfied, 31 per cent of the superintendents (35 per cent in public and 
13 per cent in separate boards) indicated there was an appeal process to follow. Forty-three 
per cent indicated there was no such process and 26 per cent were either not sure or did not 
answer. 
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Destination of Report 



In the vast majority of cases, the superintendent kept a copy of the appraisal report (87 per 
cent); a copy was usually maintained in the board's offices as well (61 per cent). In 11 per 
cent of the cases the board's ropy would be destroyed after a given number of years. 



Report Sharing 



Forty-two per cent of the 45 superintendents receiving written reports were required to sign 
them. They were not asked ^"f the results were shared with trustees, but in several case 
studies this practice was noted. In other cases, only the individual and the director were 
privy to the results. 

Fol low- Up 

Just 17 per cent of the superintendents reported that a plan was developed as a follow-up to 
their appraisal. Of these few, 67 per cent reported the plans were monitored and 92 per cent 
reported having undertaken academic or professional activities to achieve the goals of the 
plan. 

A few explict benefits resulted from the appraisal. Twenty-four per cent reported 
letters of commendation, 1 per cent a merit increment, 6 per cent reappointment to another 
term, and 22 per cent other benefits. Six per cent reported some type of negative 
consequence, though none reported two choices suggested on the questionnaire, a salary freeze 
or being placed under review. 

Ten per cent of the superintendents who reported that they had an appeal process open to 
them filed an appeal. This represented two superintendents, both of whom were with public 
boards. 

Evol ution of Pol icy 

The evolution of policy is concerned with how the policy for the appraisal of superintendents 
was developed, how it was implemented, and its degree of specification. 
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Implementdtion 



Typically, development of a policy for superintendent appraisal involved the director (88 per 
cent), superintendents (68 per cent), and, perhaps, trustees (36 per cent). Formal approval 
was required from various groups and individuals, the entire school board, 52 per cent; a 
subcommittee ot the board, 10 per cent; the director, 54 per cent; a committee of 
superintendents, 12 per cent; and all superintendents, 19 per cent. 

Nineteen per cent of the superintendents were satisfied with the process used to develop 
the policy, 51 per cent somewhat satisfied, 18 per cent somewhat dissatisfied, and 12 per cent 
very dissatisfied. The 30 per cent combined rate of dissatisfaction is one of tha highest 
noted in this study. 

Directors in 30 per cent of the boards indicated workshops had been held to assist in 
implementing the policy; half of these indicated they were not thorough. 

Degree of Specification 

Supporting documents for superintendent policies were available to 44 per cent of the 
superintendents. Sixty-two per cent of the directors indicated such documc were available 
in their board. 

Superintendents were far more critical of their ev?i"dtion policies than were tedf.hers or 
principals of their own. Fifty-one per cpnt of the superintendents indicated tha'. che 
procedures lacked detail; 29 per cent that they were not followed in practice; 19 per cent 
that they did not ensure fairness; 30 per cent that they did not help clarify roles; 37 per 
cent that there were inadequate support documents; and 14 per cent that they did not 
encourage creativity. 

laken together, one would conclude the policies are lacking in specificity. 
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Impact of Policies and Practice 



Impact is considered under three headings: compliance, effort, and the nature and degree of 
impa-t. In this instance, it was not possible to investigate which features of the evaluation 
system correlated with the impact of the system. Only 17 boards carried out formal 
superintendett appraisals, and we deemed 17 too small a number to make s.ch an analysis 
practicable. 

Degree of Compl iance 

Both superintendents and directors were asked whether or not practice followed policy. Eighty 
per cent of the directors and 59 per cent of the superintendents agreed that, for the most 
part, it did. No directors but 8 per cent of the superintendents felt practice was very 
different from policy; 33 per cent of the superintendents were not sure, 20 per cent of the 
directors felt the policy was too new to say. In addition, as noted above, 28 per cent of the 
superintendents were critical of the degree to which policy had been followed in their own 
last appraisal. 

Extent of Effort 

A number of items measured the amount of effort expended in implementing and administering 
superintendent appraisal policy in a board. The quality and length of post-conferences, noted 
earlier, are two such measures. Another is the degree of skill shown by the appraiser. 
Overall, 9 per cent of the superintendents reported their appraisers very skilful and 64 per 
cent skilful. However. 24 per cent found them not very skilful and 6 per cent not at all 
skilful. 

The directors, who for the most part were responsible for superintendent evaluation, 
reported they spent about 4 per cent of their time on superintendent appraisal, but felt they 
ought to spend about 7 per cent. Forty-six per cent placed ver> high priority on the process 
and another 46 per cent placed moderate priority on it. It was a low priority for 8 per cent. 

Trustees ranked the appraisal of superintendents as a high priority, but placed it after 
teacher, principal, and co-ordinator evaluation. 
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Taken together, one would have to conclude that, relative to the appraisal of teachers or 
principals, a modest to low amount of effort is committed to superintendent appraisal. 

Nature and Degree of Impact 

Performance appraisal can affect a person's attitudes, knowledge, or behaviour. Various 
questions assessed the impact cf superintendent appraisal on the first and last of these. 

We have already noted several of the attitudinal effects, including how superintendents 
felt at the end of the process — most felt better, implying the process provided a boost in 
morale for many. As well, 85 per cent believed that their appraiser's judgement was fair and 
80 per cent that the procedures were fair. 

As far as their performance was concerned, 9 per cent believed that it had improved a 
substantial amount, 17 per cent a modest amount, 45 per cent a slight amount, and 30 per cent 
not at all. 

Directors saw somewhat greater effect, 23 per cent reporting substantial improvement in 
their superintendents, 46 per cent modest improvement, 31 per cent a small improvement, and 
none no improvement at all. However, only 8 per cent of the directors (all of whom were in 
separate school boards) reported that their current superintendent appraisal system was very 
successful in meeting the administrative needs of the board; 92 per cent believed it was 
somewhat successful. Also, trustees, on a 5-point scale, assigned an average rating of 3.3 to 
the superintendent appraisal process as far as meeting administrative needs was concerned. 
This was a higher average rating than was given teacher, principal, or director systems. 
Their average racing for achieving the developmental needs of individuals was 3.1, about the 
same as for other evaluation plans. 

Performance Appraisal of Directors 

The director of education in each of the 30 boards in the sample was sent a director's 
questionnaire and 26 directors responded, yielding a return rate of 86.7 per cent. Of these, 
14 were from public boards and 12 from separate boards. Regions and boards represented by the 
responding directors are shown in table 83. 



Table 83: Distribution of Directors by Region and by Type of Board 



Region Public Separate 

(n-14) (n-12) 
n % ri % 



Northern 


4 


28.6 


3 


25.0 


7 


26.9 


V^atern 


2 


14.3 


2 


16.7 


4 


15.4 


Centr 


6 


42.9 


5 


41.7 


1 1 


42.3 


Eastern 


2 


14.3 


2 


16.7 


4 


15.4 


Total 


14 


53.9 


12 


46.1 


26 


100.0 



Half of the directors indicated that their board conducted a formal performance appraisal 
of the director. Appraisals were more common for directors in public boards (57 per cent) 
than in separate boards (42 per cent). For th-* most part, then, the data reported here relate 
to those 13 boards carrying out director appraisals. 

Preparation 

Preparation for appraisal includes planning, purposes, criteria, and standards as 
subcategories. 

Planning 

An assortment of methods was used to inform directors of coming appraisals. Notification in 
person (8 per cent) or by memorandum (15 per cent), an informal visit or discussion (15 per 
cent), and a reque<;t for self-evaluation (15 per cent) were all used, but a request for a 
statement about objectives, activities, and plans (23 per cent) was the most common single 
approach. At the same time, 39 per cent noted other methods were used; this category would 
include those directors who were responsible for initiating \,he process themselves. 
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Fifty- four per cent of the directors reported pre-conferences were held between 
themselves and their appraiser(s) before the process began; there was little difference 
between public and separate boards. When conferences were held, they averaged 63 minutes in 
length ( s.d. = 30 minutes) and tended to be shorter in separate boards (45 minutes) than in 
public boards (70 minut*»s). 

Eighty-five per cent of the directors (75 per cent in separate and 89 per cent in public 
boards) reported that the setting of objectives had been a central part of their last 
evaluation. Of these, 36 per cent reported that the objectives focussed on them personally, 9 
per cent on the board as a whole, and 55 per cent on both. The last response was more common 
in separate boards (67 per cent) than in public boards (50 per cent). 

Trustees were also asked a number of questions concerning the appraisal of directors. Of 
the 75 trustees returning questionnaires (38 from public and 37 from separate boards), 49 per 
cent reported their board carried out formal director appraisals. Of the 37 trustees in such 
boards, 20 had been personally involved in carrying out the last such appraisal. Of these 20, 
70 per cent reported that the appraisal had been based on the achievement of objectives. 
Fifty-seven per cent indicated the objectives focussed on both the director personally and the 
board as a whole, 21 per cent indicated only a personal focus, and the remaining 21 per cent 
only the school board as a whole. 



Purposes 



Fifty-eight per cent of the directors reported that the purposes of their last performance 
appraisal were clearly communicated to them beforehand. These purposes were to: assess the 
achievement of their objectives, 69 per cent; comply with board policy, 54 per cent; identify 
administrative strengths, 54 per cent; assess achievement of the board's objectives, 54 per 
cent; develop school/community relations, 46 per cent; identify administrative weaknesses in 
need of improvement, 39 per cent; clarify the director's role, 39 per cent; assess and 
improve curriculum, 31 per cent; reassure and develop self-confidence, 23 per cent; improve 
student learning, 23 per cent; identify in-service training needs, 23 per cent; assess 
effectiveness of instructional program, 24 per cent. The remaining choices in the 
questionnaire item were checked by 15 per cent or fewer of the respondents: to qualify the 
director for regular increment (15 per cent), to assess the effectiveness of the instructional 
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program (15 per cent), to recommend renewal of contract (8 per cent), to establish evidence 
for demotion due to inadequate administrative performance (8 per cent), and to qualify for 
merit pay (8 per cent). 

The trustees • responses followed a similar pattern, though percentages were somewhat 
higher. The top six purposes were to: assess achievement of the director's objectives (90 
per cent), assess achievement of the board's objectives (75 per cent), comply with board 
policy (70 per cent), improve student learning (65 per cent), develop school and community 
relations (65 per cent), and identify administrative weaknesses in need of improvement (65 per 
cent). The bottom three were to: qualify the director for merit pay (10 per cent), establish 
evidence for demotion due to inadequate performance (10 per cent), and qualify the director 
for a regular increment (15 per cent). 

In effect, the appraisal of directors was used almost exclusively for developmental 
purposes. 

Criteria 

The criteria that directors indicated were used and which they and trustees believe ought 
always be u'-.ed are reported in table 84. 

^^^^^ Q^- Actual ar.d Ideal Criteria Used in Director Appraisal, as Reported by Directors 

and Trustees 

Directors Trustees 

Criterion for Appraisal Actual Ideal Ideal 

^ (n-13) (n-26) (n»25) 

General domains : 



Administrative performance 


76.9? 


88. Oi 


94. 4« 


School and community relations 


53.8 


69.2 


72.9 


Program organization 


53.8 


56.0 


68.6 


Personnel 'management 


76.9 


65.4 


82.9 


Contribution to board 


76.9 


92.3 


76.8 


Religious education 


23.1 


33.3 


50.0 


Routine administration of: 








Program 


53.8 


46.2 


62.9 


Budget 


53.8 


46.2 


56.3 


Records 


23.1 


1i.7 


35.3 


Facilities 


23.1 


12.5 


26.5 


Office staff 


7.7 


25.0 


31.9 
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Table 84 (continued) 



Directors 



Criterion for Appraisal 



Actual 
(n-15) 



Ideal 
(n=26) 



Interpersonal relations with; 

Parents 46.2 

Teachers 46.2 

Principals 46.2 

Superintendents 61 .5 

Trustees 76.9 

Federation and union officials 46.2 

Knowledge of: 

Board policies 55.8 

Administrative procedures 58.5 

Relevant acts and regulations 46.2 

Personnel 50.8 

Community 55*8 

Curriculum and program 58.5 

Schools in board 58.5 

Skills: 

Management 69.2 

Supervisory 76.9 

Communication 76.9 

Organizing 61 .5 

Decision-making 69.2 

Problem-solving 61 .5 

Hunan relations 55*8 

Program evaluation 50.8 

Personnel evaluation 76.9 

Other; 

Achievement of objectives 61 .5 

Personality 22.2 



44.0 
60.4 
75.1 
73.1 
72.0 
40.0 



80.8 
72.0 
58.5 
56.0 
52.0 
40.0 
68.0 



88.5 
84.6 
92.5 
84.6 
88.5 
86.5 
88.5 
50.0 
69.2 



88.5 
44.0 



Trustees 
Ideal 
(n-25) 



59.4 
65.2 
76.1 
81 .2 
80.0 
52.9 



87.5 
81 .4 
82.6 
68.1 
62.9 
77.5 
82.9 



87.5 
91 .0 
82.9 
77.1 
90.0 
82.9 
76.8 
67.1 
70.0 



82.9 
59.4 



Overall, there 1s a fair parallel between actual and ideal factc^s. As well, trustees 
and directors appear to agree on certain areas that may be underemphas^'zed at present, such as 
the director's knowledge of and relations with those in the schools. One gets the sense that 
both parties feel that, at present, director evaluation is too much directed at the director's 
role within the board offices and not sufficiently concerned with what transpires within the 
schools. 

23 ( 



ERIC 



222 



On only one item was there a large difference between public and separate school 
directors. None of the former reported that their contribution to religious education was a 
criterion in their last appraisal; none thought It ought t. be. In contrast, 75 per cent of 
the latter reported this criterion had been used, and 67 per cent thought it ought always to 
be. Trustees' views were similarly differentiated. A bare 7 per cent of public U^rd 
trustees thought the director's contribution to religious education ought always to be a 
factor, while 89 per cent of the separate school trustees thought it should be. 

Standards 

Standards for performance on the various criteria were most often set collaboratively (42 per 
cent), although 17 per cent indicated they were set by themselves and 8 per cent by their 
appraisers. As well, one-third of the directors (all of whom were in public boards) did not 
know how they were set. 

Data Collection 

The data collection phase of performance appraisal is concerned with who provides information, 
what types of information are collected, who collects the information, and the time spent in 
the process. 

Sources of Information 

The director and trustees were most likely to provide information for a director's appraisal; 
92 per cent of the directors had done so in their last appraisal as had the trustees for 23 
per cent of them. Ninety-six per cent of the directors believed both the director and the 
trustees ought to provide information. As well, 39 per cent believed superintenuc.ts and 
principals ought to do so. Other groups some directors thought should be involved included 
teachers (23 per cent), board staff (27 per cent), and parents (8 per cent). 

Overall, it appears that directors believe too few sources of information are currently 
used in their appraisal, and that the number should be increased, particularly among 
professionals within the board. 
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Types of Information 



The types of information directors and trustees report being used in director appraisal and 
the types they believe always ought to be used are reported in table 85. 

Table 85: Actual and Ideal Types of Information Used in Director Appraisal, as Reported by 
Di rectors and Trustees 



Directors Trustees 



Type of Information Actual Ideal Actual Ideal 

(n-13) (n-26) (n-?0) (n=75) 



Objectives written by director 


84. 6^ 


92. 5^ 


85. 0? 


75. 9^ 


Objectives written by board 


25.1 


84.0 


45.0 


71 .0 


Interview by committee 


55.6 


45.8 


50.0 


46.4 


Questionnaire completed by 
superintendents or other educational 
staff supervised 


7.7 


8.7 


5.0 


23.4 


Self-evaluation questionnaire 


7.7 


42.5 


0.0 


58.1 


Written self-evaluation 


15.4 


55.5 


20.0 


58.5 


Daily calendar or diary 


15.4 


9.5 


0.0 


19.0 


Standardized test scores for students 


0.0 


0.0 


0.0 


15.1 


Oral report 


61.5 


50.4 


50.0 


57.5 


Reports for which the director 
had been responsible 


46.2 


40.0 


50.0 


50.0 


Report from previous appraisal 


30.8 


54.2 


25.0 


50.0 


Reports on achjevement of objectives 


53.8 


88.0 


55.0 


80.5 


Results of external assessment 
of board 


0.0 


54.8 


5.0 


55.9 


Goal package the director had 
prepared for self and board 


61.5 


70.8 


50.0 


59.7 
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The data suggest that a relatively broad array of information is currently used in the 
assessment of directors, including lists of objectives, interviews, oral reports, and reports 
on the achievement of objectives. At the same time, both directors and trustees appear to 
agree that the data base should be increased to include, on a more regular basis, objectives 
written by the board, sM f-evaluation by the directors, and written reports from previous 
appraisals. 

The ^-esults parallel, to a degree, those for superintendents in that a desire is implied 
for more direction — and perhaps roie clarification from Lheir board. Then, directors 
wish to be judged on results, particularly results as reflected in reports that address the 
achievement of their written objectives. 

Collectors of Information 

The director and the trustees were the sole collectors of information for the appraisal of all 
the directors in the sample. In 73 per cent of the cases, the appraisal was carried out by a 
team of trustees. Through interviews we learned that in some cases the appraisals were 
conducted by the entire board. 

The only other party ever involved in some way in the process of appraising directors was 
the superintendent, who participated in only 23 per cent of the appraisals. 

Time Spent 

Seventy-five per cent of the directors reported less than one day had been spent in collecting 
information for their last appraisal; yet, 17 per cent indicated more than five days were 
spent. The remaining 8 per cent indicated one day was spent. Apparently, the process of 
evaluating a director by a team of trustees is very different from the appraisals carried out 
for teachers and principals. In interviews it b'^came clear that many directors were the ^orce 
behind their own appraisal and that they linked the process to the setting and meeting of 
objectives for the board as a whole. 
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Reporting and FoUowUp 



The nature of the reporting, including any form completed and filed, the sharing of this 
document, anc -ne steps taken after the completion of a performance appraisal, are the topics 
of this section. 



Nature of Report 



Post-conferences, reported by 54 per cent of the directors, were as common as pre-conf erences. 
They were not held for half of the directors of public boards, but were held for all directors 
of separate boards. Fifteen per cent of the directors did not answer this question. 

In the majority of cases (86 per cent), a subcommittee of trustees was present with the 
director at the post-conference. Twenty-nine per cent of the directors reported all trustees 
were present. In public boards, someone else may have been present; the questionnaire did 
not ask wno this other person, or persons, might be. On average, the conferences lasted 74 
minutes ( s.d. = 25 minutes); there was little difference between separate and public boards. 
Reports of trustees as to the length of these conferences agreed with those of the directors. 

Assessment by directors of the quality of the post-conference was mixed: 14 per cent 
reported that it was very well planned, 71 per cent fairly well planned, and 14 per cent 
poorly planned. Free participation by all parties was reportad by 57 per cent of the 
evaluatees; 14 per cent reported both were restrained; 14 per cent that the appraiser was 
involved and the appraisee was restrained; and 14 per cent that the appraisee was involved 
and the appraiser restrained. The experience wes not at all threatening for 86 per cent of 
the directors while 14 per cent found it somewhat threatening. 

Trustees had somewhat different perceptions: 46 per cent reported very well planned 
post-appraisal conferences, 46 per cent fairly well planned, and 9 per cent poorly planned. 
All believed there was free and open participation by both parties. 

Directors' assessment of post-conferences was provided by only seven directors four 
public and three separate. The small number of responses comes from the small number of 
directors for whom such conferences were held. At any rate, four directors reported sincere 
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praise had been given during the conference; three reported no praise was given. Praise was 
reported by all separate school directors but by only one of the public school directors. 
Constructive criticism was provided in three cases, partly constructive in three, ind no 
criticism in one. All separate school directors reported all criticism was constructive; no 
public school directors did so (three reported it partly constructive and one said none was 
given). 

After the post-conference, two of the directors felt good and four felt neither good nor 
bad. One felt somewhat negatively. Two of the separate school directors felt good, but none 
of the public school directors reacted in this way. 

Written reports were provided to 39 per cent of the directors appraised; 54 per cent 
reported that they were not provided, and 8 per cent did not respond. All were satisfied with 
the type of the report used; none were dissatisfied and none were very Satisfied. Thirty per 
cent of the trustees were very satisfied, though, and another 50 per cent were satisfied. 
Twenty per cent of the trustees reported being very dissatisfied. 

Destination of Report 

In the majority of cases, the director kept a copy of the appraisal report (60 per cent); a 
copy was usually maintained in the board's offices as well (60 per cent). In 20 per cent of 
the cases the board's copy would be destroyed after a given number of years. 

Report Sharing 

Directors were asked if they were -equired to sign their report; only four directors 
responded, with just one stating there was such a requirement. These numbers were so small 
since only five of the 13 directors who had been appraised recently had written reports in the 
first place. Directors were not asked specifically if the results were shared with all 
trustees, but, since the appraisals were typically carriea out by a team of trustees, this 
would probably have been the case in most instances. 
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Follow-Up 



Just one director reported that a plan was developed as a follow-up to appraisal. This 
director indicated that the plan was not mon, cored and that no academic or professional 
activities were undertaken to achieve the goals of the plan. This absence of plans was 
confirmed by trustees. 

A few explicit benefits resulted from the appraisal. Two directors reported receiving a 
merit increment and four other benefits. None reported any negative consequences. 

Two directors reported that they had an appeal process open to them and seven that they 
did not; four did not respond. Neither director with the option available chose to appeal. 

Evolution of Policy 

The evolution of policy is concerned with how the policy for the appraisal of directors was 
developed, how it was implemented, and its degree of specification. 

Implementation 

Typically, development of a policy for the performance appraisal of a director involved the 
director (92 per cent) and trustees (92 per cent). Formal approval was usually required 
from the entire school board (77 per cent), often with the director's consent also being 
required (31 per cent). 

Forty-two per cent of the directors were satisfied with the process used to develop the 
policy, 39 per cent somewhat satisfied, 8 per cent somewhat dissatisfied, anH 8 per cent very 
dissatisfied. 

Trustee reports of the development of policies for director appraisal were somewhat 
different. In addition to involvement of trustees (92 per cent) and the director (62 per 
cent), 8 per cent reported involvement of superintendents and 11 per cent outside consultants. 
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Forty-two per cent of the trustees were very satisfied with the process used and 33 per cent 
somewhat satisfied; 11 per cent and 14 per cent were, respectively, somewhat dissatisfied and 
very dissatisfied. 



Degree of Specification 

Sixtytwo per cent of the directors reported that their appraisal policy was a public document 
approved by the board; only 8 per cent reported it was part of a confidential document. 
Thirty-one ^i^r cent checked "other". 

Both dinictors (39 per cent) and trustees (35 per cent) were critical of the lack of 
detail in the director policy in their board. A significant number of the directors (23 per 
cent) also felt the procedures did ►^ot ensure fairness. Trustees were likely to report that 
supporting documents were inadequate (16 per cent). 

Impact of Policies and Practice 

Impact is considered under three headings: complia-Ke, effort, and the nature and degree of 
impact. In this instance, it was not possible to investigate which features of different 
evaluation systems correlated with the impact of the system. Only 13 boards carried out 
formal director appraisals, and we deemed 13 too small a number to make such an analysis 
practicable. 

Degree of Compliance 

Both directors and trustees were asked whether or not practice followed policy. Thirty-nine 
per cent of the directors indicated practice v/as as described in policy and another 39 per 
cent approximately as described; 23 per cent were not sure. Fifty-seven per cent of the 
trustees believed policy was followed; 26 per cent believed practice was approximately as in 
policy; and 3 per cent felt it was very different. Fourteen per cent were not sure. 




Extent of Effort 



A number of items measured the amount of c^ffort expended in imple.Tienting and administering 
appraisal policie*- for directors. The quality and length of the post-conference, noted 
earlier, are two such measures. Another is the degree of skill shown by the appraisers. No 
directors reported that their appraisers were very skilful and just 20 per cent reported that 
they were skilf' 1. The majority, 70 per cent, found them not very skilful and 10 per cent not 
at all skilful. 

In spite of the*^; relative lack 0"^ <^kill, trustees apparently took the process seriously. 
Forty-six per cent of the directors reported that the trustees carrying out th? appraisal were 
very serious; the balance, 54 per cent, saw them as being somewhat serious. Trustees 
responses were: very seriously, 74 per cent; somewhat seriously, 16 per cent; not seriously 
at all, 5 per cent; not sure, 5 per cent. As well, on a S-point scale, trustees rated the 
evaluation of the director as having hig importance, 4.3. Still, this was the lowest ranking 
given any category of staff. 

Taken together, one would have to conclude that, relative to the appraisal of teachers or 
principals, a modest to low amount of effort is committed to the appraisal of directors. 

Nature and Degree of Impact 

Performance appraisal can affect a person's attitudes, knowledge, or behaviour. Various 
questions assessed the impact of director appraisal on the first and last of these 

Already noted have been several of the attitudinal effects, including how directors felt 
at the end of the process most felt neither better nor worse, implying the process had 
little effect on their attitudes toward their jobs. Although it did not boost thpir morale, 
it did nnt weaken it. At the sanre time, 80 per cent of the directors felt the judgement of 
their appraisers ana the procedures were fair. 

As far as their performance was concerned, none of the directors reported substantial 
improvement, 44 per cent modest improvement, 33 per rent slight improvement, and 22 per cent 
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none at all. Trustees' respons- ; were slightly less positive, 23 per cent reporting modest 
improvement in their director's performance, 46 per cent slight improvement, and 31 per cent 
none at al 1 . 



Also, trustees, on a 5-point scale, assigned an average rating of 3.1 to the process of 
appraising the director as far as meeting administrative needs was concerned. This was a 
lower average than was given principal or superintendent appraisal systems but was higher than 
the 3.0 average given teacher systems. The trustees' average r?' ng for achieving the 
developmental needs of individuals was 3.0 for the director's appraisal, which was the lowest 
rating of all the appraisal systems. 

Demographic Profile of ELS Respondents 

Table 86 provid3S the age distribution of the respondents by the role of the respondent. As 
one would expect, age tends to increase with the level of the position. Trustees' ages, 
however, reveal somewhat greater variability than do the ages of professionals within the 
system. Not reflected in the table are differences between public and separate schools. 
Staff in the latter are, on average, somewhat young. ? . For example, 23 per cent of the 
separate school teachers are 30 or under as compared with only 12 per cent of public school 
teachers. Similarly, 10 per cent of the separate school principals are 35 or under as 
compared with 2 per cent of public e'lerr.entary school principals. 

Table 86: Age Distribution by Role of Respondent 



Age 


Teacher 


Principals 


Supfs. 


Directors 


Trustees 


in Years 


(n-4040) 


(n-876) 


(n-113) 


(n-26) 


(n-75) 


20 to 25 


3.2? 


0.0 


0.0 


0.0 


0.0 


26 to 30 


12.5 


0.3 


0.0 


0.0 


1.3 


31 to 35 


24.7 


4.5 


2.7 


0.0 


12.0 


36 to 40 


23.7 


15.9 


7.1 


3.8 


10.7 


41 to 45 


15.0 


33.7 


19.5 


11 .5 


24.0 


46 to 50 


9.2 


19-9 


30.1 


26.9 


• 14.7 


51 to 55 


6.9 


18.2 


23 9 


46.2 


20.0 


56 to 60 


3.3 


6-7 


15.0 


1.1 


6.7 


61 to 65 


1 .4 


0.9 


J .8 


3.8 


8.0 


over 65 


0.0 


0.0 


D.O 


0.0 


2.7 
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The trustees responding had an average of 7.5 years of experience as a trustee. The 
separate school board members were slightly more experienced, averaging eight years, than were 
public board trustees, who averaged seven years. No other information was collected about 
trustees. 

The teachers in the sample were distributed as follows: classroom teacher, 70 per cent; 
department head in secondary or junior high school, 9 per cent (24 per cent among secondary 
respondents); elementary specialist teacher, 13 per cent (20 per cent among elementary 
respondents); vice-principal with teaching responsibilities, 1 per cent; counsellor with 
teaching responsibilities, 1 per cent; and other, 5 per cent. Forty-five per cent of the 
teachers taught all subjects, 53 per cent specialized, and 3 per cent checked o^her. 
Specialization was, as would be expected, most common at the secondary level, where 94 per 
cent of the teachers indicated they did so; at the elementary level, only 30 per cent of the 
teachers did so. 

Distribution of principals was half in public elementary schools, 12 per cent in 
seconda^^y schools, and 37 per cent in separate schools. 

Thirty-seven per cent of the superintendents were classed simply as "superintendent", 
wnile 28 per cent were area superintendents, 6 per cent assistant superintendents, 6 per cent 
superintendents of program, 3 per cent superintendents of personnel, 6 per cent 
superintendents of business, 5 per cent assistant or associate directors, and 9 per cent 
"other" superintendents. 

Table 87 reports the academic qualifications of the teachers, principals, 
superintendents, and directors. 
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Table 87: Highest Academic Qualification of Respondents by Role of Respondent 

Highest Teachers Principals Supt ' s . Directors 

Qualification (n-4028) (n-878) (n-113) (n-26) 



Teachers' college, 
no degree 


19. 4^ 


1 .0% 


0.9% 


0.0^ 


B.A. or B.Sc. 


59.7 


2Q ."S 




1 Q 0 


M.Ed., M.A., 
M.Sc. M.S. A* 


12.7 


67.5 


76.1 


73.1 


Ed.D. or Ph.D. 


0.4 


0.5 


3.5 


1.1 


Other 


1^1 


1 .5 


5.3 


0.0 



Teachers without degrees were concentrated at the elementary level, where 26 per cent of 
the teachers were in this category; only 8 per cent of the secondary teachers were without 
degrees. 

Among all teache»^s, 77 per cent reported specialized, rather than general, bachelor's 
degrees. The percentage was higher among secondary (84 per cent) than among elementary (71 
per cent) teachers. Specialized degrees were also reported by 68 per cent of the principals 
(87 per cent at the secondary and 66 per cent at the elementary level), 72 per cenv of the 
superintendents, and 50 per cent of the directors. General bachelor's degrees were much more 
common among separate school directors (75 per cent) than among public school directors (29 
per cent). 

The years of experience of the various groups of educators are reported in table 88. 
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Table 88: Experience of Respondents by Role of Respondent 



Experience Teachers Principals Supt*s. Directors 

(n«4022) (n-874) (n-112) (n-26) 



Years 


in 


position 


7.41 


4.61 


6.09 


6.85 


Years 


in 


role 


13.80 


12.41 


7.93 




Years 


in 


board 


11 .44 


18.96 


16.18 


11 .96 


Years 


as 


educator 


13.80 


23.48 


25-68 


29.04 



Variations in experience do occur among different types of school. For example, 
secondary teachers report an average of 9.8 years in their present school, public elementary 
teachers 6.7 years, and separate elementary teachers 5.4 years. Cverall, teachers in public 
boards of education report 11. S years of experience while those in separate boards report 10.2 
years. Principals in public boards averaged 20.7 years (.elementary) and 19.3 years 
(secondary), whereas separate school principals averaged 16.6 years. Public board directors 
also have, on average, more experience in their boards (13.6 years) than do separate school 
directors (9.4 years); the same holds for superintendents (26.4 years vs. 24.1 years). Yet, 
public superintendents tend to be newer to their positions (7.4 years) than is the case in 
separate schools (9.2 years). All these age differences can be explained by the different 
timing of growth in public and separate school enrolments. The latter have increased 
enrolment more recently and have generally experienced less enrolment decline. Hence, their 
staffs tend to be younger, with administrative promotioii also occurring at a younger age. 
Finally, it is worth noting that the decline in average experience within a school board 
associated with superintendents and, particularly, directors reflects movement between boards 
of officials at these levels. 

Overall, 40.3 per cent of the teachers were male. In public elementary schools the 
percentage was 26.8, in secondary 65.7, and in separate 25.4. In addition, 87.7 per cent of 
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the principals were male — in public elementary, SI. 2 per cent, in secondary, 98.1 per cent, 
and in separate, 79.4 per cent. Among superintendents, 96.5 per cent were male. All separate 
school superinteodents were men, as were 94.9 per cent of those in public boards. All of the 
directors were men. 

Permanent contracts were held by 92.5 per cent of the teachers; 6.2 per cent had 
probationary contracts (9.6 per cent in separate and 5 per cent in public schools); 0.1 per 
cent had letters of standing; and 1.2 per cent other forms of contracts. 

The vast majority (93.6 per cent) of the principals had permanen!" appointments. As well, 
2.3 per cent had probationary appointments to the position, 0.8 per cent were acting 
principals, 2.5 per cent held term appointments, and 0.8 per cent had some other arrangement. 

Term contracts were more prevalent at the higher levels. Of the superintendents, 81.4 
per cent had permanent appointments and 15.9 per cent term contracts, with 2.7 per cent 
reporting "other". Of the directors, 69.2 per cent had permanent appointments and 23.1 per 
cent term contracts, including 35.7 per cent of those ir. public boards. As well, one director- 
{3.8 per cent) reported a probationary contract and one director indicated "other". 

The average size of school reported by principals was 345 for public elementary, 410 for 
separate, and 945 for public secondary schools. The average amount of time spent teaching by 
principals was 12 per cent; it ranged from 2 per cent at the secondary level to 14 per cent 
in public elementary schools. 

Superintendents on average supervised eight office staff, 19 principals, 395 teachers, 
four assistant adm-^nistrative staff, and four others. The average number of schools 
supervised was 20; the average was higher in public boards (25) than in separate boards (12), 
a difference no doubt explained by the larger sizes of public boards. 

Directors on average were responsible for 32 office staff, five superintendents, three 
assistant administrative staff, and 60 others. 

The percentages of the four groups that had experienced performance appraisals are 
reported in table 89. 
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Table 89: Appraisals of Respondents by Year and by Role of Respondent 



Year 


Teachers 


Principals 


Supt's. 


Directors 




(n-40ej2) 


(n-679) 


(n»1 14) 


(n-26) 








25 ,4i 


34.6* 






18.7 


35.1 


1 1 .5 


1 Q An n 1 
1 you^o 1 




7.3 


1 .8 


0.0 


1 979-80 


6.7 


5.6 


0.9 


3.8 


1978-79 or before 


12.7 


11.3 


2.6 


7.7 


Have not been 
formally evaluated 


5.2 


23.9 


33.3 


42.3 


Not answered 


20 


1.6 


0.9 


0.0 



The data in table 89 suggest, as did the detailed review of the questionnaire results, 
that formal performance appraisal is most common at the classroom level and least common at 
the director's level. As well, the data suggest a trend to more evaluation in recent years, 
though those who had been more recently appraised may have been more likely to return 
questionnaires. 

Directors were asked if statistics were kept on teachers' ratings as established by 
formal evaluations. One director (3.8 per cent) answered affirmatively; the remainder 
indicated that none were maintained or that the question was not applicable. 

Terminations and Grievances 

Some quantitative data were collected from directors concerning the numbers of staff who had 
been placed under review, were dismissed, or who filed some type of action (appeal, grievanc 
or lawsuit) against the board as a result of performance appraisals. Their responses are 
summarized in tables 90 to 92. 
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^I'^^l^^^"^ standard Deviation s of Staff Under Review Between September 1981 and May 
1983 by Staff Category and by Type of Board, as Reported by pTTictofi 

Staff Category PubUc Separate Total 





(n=14) 


(n-12) 


(n«26) 


Teachers 


5.67 


4.42 


5.04 




(5.05) 


(4.42) 


(5.21) 


Principals 


1.77 


0.60 


1 .26 




(1-09) 


(0.84) 


(1.14) 


Superintendents 


0.17 


0.00 


0. 10 




(0.39) 


(0.00) 


(0.30) 


Other 


1.14 


1 .29 


1 .21 




(2.19) 


(2.14) 


(2.08) 



a 



Standard deviation in parentheses. 



Table CI 



n^i^^^. 'u Devi ations of Staff Terminated for Unsatisfactory Performance 

Bg^^en_j^ ^ 1981 and May 1983 by Staff Category and b y Type of Board, as Reported br 



Staff Category 



Public 
(n-14) 



Separate 
(n«12) 



Total 
(n-26) 



Teachers 3.75 1.75 2.75 

(4.56) (1 .14) (3.40) 

Principals 0.58 0.00 0.33 

(0.79) (0.00) (0.66) 

Superintendents 0.00 0.00 0.00 

(0.00) (0.00) (0.00) 



Standard deviation in parentheses. 



ERIC 



25, 



237 



Table 92: Means and Standard Deviations of Actions Filed Against Boards Between September 
1981 and May 1983 by Type of Action and by Type of Board, as Reported by Directors 



Type of Action 



Public 



Separate 

(n=12) 



Total 



(n=14) 



(n=26) 



Appeals 



0.58 
(0.79) 



0.78 
(1 .99) 



0.67 
(1.39) 



Grievances 



0.33 
(0.89) 



0.56 
(1.33) 



0.43 
(1 .08) 



Lawsuits 



0.17 
(0.39) 



0.20 
(0.42) 



0.18 
(0.18) 



Standard deviation in parentheses. 

All the averaqe? in tables 90 to 92 are quite low. It appears that, in a typical school 
board, between one and two teachers are placed under review in a given year. On average, 
placing a principal under review occurs once in two years. Negative ratings of 
superintendents are virtually non-existent. Terminations as a result of unsatisfactory 
performance are also rare, occurring about once per year in a typical board for teachers, and 
once in five to 10 years for principals. Appeals, grievances, and lawsuits as a result of 
performance appraisals appear to occur in the typical board with a frequency of once ev^ry 
five years in the first instance to once in 15 years in the last instance, assuming a uniform 
distribution. 

It was not clear whether the number of staff placed under review, dismissed, or who filed 
appeals, grievances, and lawsuits, were hallmarks of an effective or ineffective evaluation 
system. One could argue a good system would result in the review and dismissal of more 
ineffective staff members or one could claim an effective system would help staff improve and 
would therefore result in fewer such actions. 

To shed light on this issue, the numbers of teachers who had (1) been placed under 
review, (2) been dismissed, (3) filed appeals, (4) filed grievances, and (5) filed lawsuits 
were correlated on a board by board basis with each of the variables treated in the analysis 
of the effects of different evaluation polic'^s. This analysis was conducted only for 
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teachers since only for them was the frequency of such events considered sufficient for 
statistical analysis. Correlations were considered significant if the £ value was less than 
or equal to 0.02. 



Of all correlations, the highest were between the number of appeals and grievances filed 
and the presence of a formal system to review the appraisal policy; the correlations were 0.71 
and 0.72, respectively. We take this relationship to mean that boards which had suffered 
these problems instituted formal reviews of their evaluation policies in order to determine 
what problems were behind the large number of actions being filed. 

Most other significant correlations were negative; for example, the number of staff 
placed under review had the following correlations: with the percentage of teachers who had 
read the evaluation documents, -0.42; with the percentage of teachers reporting students had 
provided information, -0.47; with the extent teachers see the evaluation system achieving its 
goals, -0.47; with a final post-evaluation conference, -0.50; with development of a plan, 
-0.59; and with how seriously their evaluator had taken the process, -0.60. We take these 
correlations to mean that fewer teachers are placed under review when evaluation systems are 
most effective at achieving their goals, and especially their developmental goals. 

The number of teachers dismissed also had negative correlations with a number of 
appraisal characteristics: with the ex ent of principal involvement in their last appraisal, 
-0.52; with development of a plan, -0.47; with how seriously their appraiser took the process, 
-0.51. As well, the number of appeals and grievances had correlations of -0.48 and -0.44 with 
the number of post-observation conferences. 

There were only two significant correlations between any of these variables and the 
demographic variables: the percentage of teachers placed under review was positively 
correlated with the average number of years teachers had been in the board, and the number of 
lawsuits filed was positively correlated with the percentage of teachers on probation. The 
number of lawsuits, incidentally, was negatively correlated with the rate of enrolment decline 
in a board, meaning that *.hey were more likely in growing or stable boards than in those where 
enrolment was declining. This is consistent with the positive correlation with the percentage 
of probationary teachers. 
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In conclusion, it appears that actions of the sort reported in tables 90 to 92 are 
associated with poor appraisal systems; good appraisal systems apparently resolve problems 
raised by poor performance, improving the performance of teachers or providing them with 
sufficient valid and reliable information to convince them that an alternative career would be 
more suitable. 



The purpose of this summary is to bring together and compare information concerning the 
different phases of the performance appraisal systems as they apply to teachers, principals, 
supe'^intendents, and directors. 



Planning for evaluation varied considerably among the four groups. It was most thorough with 
the principals, most of whom were personally informed of the impending evaluation and had 
lengthy pre-conferences with their appraisers to review the process, critieria, and purposes 
of their appraisal. As well, most were asked to provide a statement about their school, and 
wor^ with their appraisers to set objectives for themselves and their school whose 
achievement could be assessed in the appraisal. While most teachers were also informed of 
coming evaluations personal Iv, superintendents were not, and both of these groups were far 
less likely to report havincj had pre-conferences. Teachers were far less likely to have been 
involved in setting objectives than were principals or superintendents. When objectives were 
set for teachers, however, they tended to be set collaboratively and focussed on overall 
performance, not just their work in the classroom. For directors, notification might come in 
any one of several ways, but many apparently initiated the process themselves. 
Pre-conferences were common, but by no means universal, for directors and objective-setting 
was the norm. 

In the analyses of the impact of various appraisal practices carried out for teachers and 
principals (there were insufficient data to conduct similar analyses for superintendents and 
directors), the holding of pre-conferences, the length of pre-conferences, and the use of 
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objective-setting were strongly related to positive results of evaluation. At the same time, 
the lack of these practices was evident in the appraisal of superintendents, the group which 
reported the least satisfying and least effective appraisal experiences. 



The pu-poses of evaluation reported by all four groups were concerned with developmental 
rather than administrative needs. Devriopmental needs include matters such as the improvement 
of staff performance and the improvement of the educational program whi^ administrative needs 
include the identification of individuals for promotion, demotion, and merit pay. As well, 
all yroups supported the emphasis on developmental purposes. When asKed if they believed 
separate administrative and developmental appraisal systems for teachers should be used, all 
groups were rather evenly split; many were not certain such a separation was feasible. 

A few administrative uses of appraisal were reported, though these were expected. Among 
teachers, use of evaluation to make recommendations for permanent contracts was universal. 
Among administrators, and particularly among superintendents and directors, use of performance 
appraisal to assess their achievement of objectives was the norm. 

Over two-thirds of all groups except the directors reported that the purposes of their 
last appraisal had been clearly communicated to them. In analyses relating the method of 
appraisal to the effectiveness of appraisal, clear communication of the purposeb of evaluation 
stood out as a variable of exceptional importance. 

Criteria used in appraisal reflect the expectations evaluatees and evaluators have for a 
given role. In effect, criteria provide a job description. Overall, there was a high degree 
of consensus concerning criteria at all levels, i.e., evaluators and evaluatees generally 
agreed on what criteria were used in practice and on what ought to be used. 

However, it is still worth noting where minor divergences in opinion occurred. Classroom 
teachers, for one, would like less emphasis placed on their out-of-classroom activities than 
is the case at present, while their evaluators feel that a broader conception of the role, 
including such matters as community/school relations, is important. For themselves, 
principals would like to see less emphasis on their role as disciplinarian while teachers 
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would like more of this emphasis. Also, principals believe a slight increase in emphasis on 
their role in program development would be appropriate, while superintendents would prefer a 
moderate increase -- and teachers a moderate decrease in this emphasis. 

Superintendents, though generally supportive of the criteria used in their appraisal, 
believed that more emphasis should be placed on their knowledge of board pol''"ies and relevant 
acts and regulations. For directors, trustees and directors concurred that a similar increase 
in emphasis on the knowledge of the legal aspects of the job was needed, along with more 
emphasis on the directors* knowledge of the program and schools in their system. 

The criteria used did not seem particularly relevant to the improvement perceived on the 
part of teachers or principals. What trends there were suggested emphasizing classroom 
activities was associated with greater development on the part of teachers, while emphasizing 
broader activities such as th Jr contribution to the board was most productive in the case of 
principals. Put another way, criteria seem important not in determining how well work is 
done, but in determining what work is done. Clearly, use of performance appraisal to steer 
school systems is prevalent and may be of considerable importance in helping school boards 
achieve their objectives. This use of criteria was particularly evident in boards with a 
significant number of French-language schools. In these, the use of criteria concerned with 
the staff's efforts to preserve the French language and culture was evident. Nevertheless, 
there is little evidence that the use of performance appraisals to direct a school board's 
activities is widely perceived. 

While criteria relate to what is assessed, standards are concerned with the level of 
performance on a givon criteria. Teachers were most likely to believe that standards were set 
by their appraisers, though a minority believed they were stated in their board policy. Given 
the difficulty of specifying standards in complex matters requiring professional judgement, 
such as classroom management, the former view is probably the more realistic of the two. 

For principals ^nd directors, the collaborative approach to setting standards was most 
often mentioned. This method parallels the use of objective-setting, an important part of 
which is the specification of how the achievement of an objective is to be assessed. The 




ERIC 



242 



largest percentage of superintendents, however, stated that their appraiser set the standards. 
As well, many superintendents did not know how standards were set, a finding that again 
reflects poorly on the current practice of appraising superintendents. 

Data Col lection 

The primary sources of information for the evaluation of staff were the evaluatee and the 
evaluator; this generalization holds for all four groups. The involvement of others is most 
tommon in the case of principal appraisal; often, teachers will be asked questions by a 
principal's appraiser(s). In a few cases, a similar process is followed in teacher 
evaluation, with the principal speaking with students c-bcut classroom activities. The 
provision of information by others in the apf/aisal of superintendents and directors virtually 
never occurred, though superintendents were of the opinion that principals ought to be 
involved and, similarly, director's believed that ojperintendents ought to be when the director 
was appraised. 

Analysis of the impact of various appraisal practices sur^ests that involvement of 
individuals other than the appraiser and appraisee is important. For teachers, the provision 
of information by students was positively related to the results of evaluation; fo.^ 
principals, the provision of information by teachers was similarly related. 

Specific notes taken by appraisers are the most favoured type of information among 
teachers and principals; reports on achievement of their objectives are the most favoured 
among superintendents and directors. All believe more use should be made of self-evaluation 
questionnaires, and most believe less use should be made of materials produced in the course 
of their jobs - lesson plans, calendars, budgets, and the like. Finally, there is a 
consensus that standardized tests, records of student absenteeism, and other quantitative 
measures of output ought not be used in staff appraisals. 

Use of specific and, to a degree, general notes made by appraisers were, in fact, 
positively related to the impact of the appraisal of both teachers and principal?. As well, 
use of observation and interviews in teacher appraisal was associated with more effective 
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appraisals. One type of information, student absenteeisiL records, was negatively associated 
with the benefits of teacher evaluation. Results for the use of other types of informa'"ion 
were mixed or negligib'.v. 

Checklists, one form of recordir.g informatioi often associated with forral appraisal 
systems, were neither positively nor negatively related to the results of the evaluation of 
teachers or principals. Though the use of such instruments is not particularly widespread 
(they were reported by about 30 per cent of the teachers and 20 per cent of the principals), 
most respondents would prefer that they be used somewhat less than at present. 

For most teachers, principr's, and superintendents, a single individual was primarily 
responsible for their appraisal and collected all information. A significant minority of 
principals, however, were appraised by a team that visited their school and collected the 
data. The team approach was standard in the appraisal of directors; most often, a 
subcommittee of trustees took the responsibility. Also, in some cases, teachers reported that 
their superintendent took part in their appraisal; such a practice is most evident in special 
cases, as when a teacher is on a probationary contract or is under review, although in some 
boards superintendents evaluate all teachers. Finally, appraisers from outside the school 
system were not used for any appraisal. 

There is some evide' ce that a superintendent's involvement in teacher evaluation helps to 
make teachers take the process more seriously. As well, there was an indication that team 
appraisals of principals were more effective than individual appraisals. 

For teachers, the time spent to collect information was measured by the number of 
observations, with three or four being most common. For principals, the time spent was 
measured in days, with one two, or three being most common. Both groups thought that more 
time should be spent, and the evaluators of both stated that more time was in fact spent than 
was reported by the evaluatees. As well, post-observation conferences for teachers, reported 
most frequently after every observation, typicjlly lasted about 20 minutes. The time spent in 
the collection of information and in conferring with the evaluatee both proved important 
variables in explaining the effectiveness of teacher and principal evaluation. 
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In contrast, most superintendents and directors reported less than one day was spent in 
the collection of information for their appraisal, though significant numbers reported more 
than five days were used. Perhaps the evaluation process is less of an event and more of a 
continuing activity for senio> administrators, with regular reports on the achievement of 
objectives and the like being placed on file throughout the year. If this is not the case, 
then one would have to conclude little time is devoted to he appraisal of the most senior 



staff. 



Reporting and Fol low-Up 

Post-evaluation conferences at which the results of the entire process were presented to and 
discussed with the evaluatees varied in frequency a. the different levels. They were most 
frequent at the p incipal and superintendent levels, and least common at the director and 
teacher lavels. Judging from the responses to questions concerning the evaluatees' reactions 
to these conferences, those for teachers and principals were best planned and carried out. 
And, while teachers, principals, and superintendents reported the conferences provided a boost 
in their morale, most directors reacted neither positively nor negatively. 

The existence of post-conferences for teachers (both after every observation and at the 
end of the process) and for principals was p sitively related to the effectiveness of the 
appraisal process. The length of s;h conferences is also important. 

The most common type of report form at all levels was a report under several headings. 
The next most popular was an unstructured report. Ratings, of either specifir activities or 
of the quality of one's overall work, were used in relatively few cases. In any case, there 
was little evidence that the form of the report made much difference, though the use of 
ratings of activities for principals w,s negatively related to how seriously they took the 
process. 

Processes to appeal one's evaluation report were reported by less than half the 
respondents at 11 levels. The axistGnce of such systems appeared important to the 
effectiveness of teacher evaluation, b.t not to that of principal. Many teachers, principals, 
and superintendents were unsure if an appeal process was available to them or not. 
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In almost all cases, one copy or an evaluation report was kept by the evaluatee and one 
was sent to the school board's office where it became part of the permanent record of the 
evaluatee. The practice of destroying evaluations after a certain number of years was rare. 

Evaluation reports tended to be private matters between the evaluator and evaluatee 
Exceptions occurred in the case of some principals, parts of whose assessment were shared 
either by the principal or the evaluator wivh other members of the school staff In such 
cases, evaluative information concerning the principal was omitted, with the focus being on 
the quality of the school and opportunities for ^morovements. Also, for some superintendents, 
eveluatioRS were shared with school trustees; this was also the case with most directors. 

Follow-up to evaluation was more evident by its omission than its presence. Plans to 
implement suggestions made in their a-^oraisal were reported by less than 30 per cent of the 
respondents in all staff categories; just 7 per cent of the directo-s reported such ans. 
Monitoring of plans that were made occurred about half the time, as did engagement in 
professional or academic activities to fulfil the plan's objectives. Positive feedback in the 
form of letters of commendation were relatively rare. They were most com.non among 
superintendents, a quarter of whom reported such encouragement. Another form of positive 
feedback, merit pay, was reported by 15 per cent of the directors; it was virtually 
non-existent at other levels. Negative sanctions in the form of being placed under review or 
having one's salary frozen were also very r^are. 

In spite of the lack of follow-through on plans, the making of plans did seem important 
to the effectiveness of teacher and principal evaluation. In boards where such plans were 
common, appraisal systems were more likely to be perceived as being effective in improving the 
quality of work. In contrast, monitoring plans or providing letters of commendation were not 
significantly related to an evaluation system's effectiveness. 

Evol ution of Pol icy 

Policies for performance appraisal of all staff were developed in various ways. 
Superintendents, principals, and teachers (in declining order of frequency) were likely to be 
involved in developing teacher policies; superintendents and principals were most likely to 
participate in the development of principal policies; and dirpctors alone usually developed 
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superintendent policies, Approval for teacher and principal policies were granted by the 
senior administration, generally, though school boards were sometimes involved. School boards 
generally approved superintendent and director policies. 



Teachers and principals were generally satisfied with the procedures used to develop 
policies, though each would prefer greater participation on the part of their own 
constituency. As well, principals felt teachers could play a greater role in the development 
of principal policies. 

Directors and, particularly, superintendents were less satisfied with the process by 
which their policies were developed. For superintendents, :he process v-as apparently too 
unilateral. 



How a policy was developed was related to its effectiveness. Boards which involved 
teachers and principals in the development of their teacher policies were mor^ likely to have 
effective systems; similarly, in • ^ ^ment of trustees, teachers, and principals tended to be 
related to more effective principal policies. 

Supporting documents describing the appraisal system and workshops were often provided to 
assist in implementing policies. Documents on teacher evaluation were nearly universal; 
documentation for principal appraisal also tended to be available. However, supporting 
materials were far less likely for superintendent and director evaluation. As far as 
workshops are concerned, workshops for principals on the topic of teacher evaluation were most 
common, evident in about 80 per cent of the boards, workshops for teachers on this topic were 
not widespread, however. Principals and superintendents were equally likely to report 
workshops on principal evaluation; tnese occurred in about half the boards. Workshops 
concerned with superintendent appraisal were held in about one-third of the boards. 

For teacher evaluation, the existence of supporting documents and workshops was related 
to the perceived effectiveness of the appraisal system; similar, albeit weaker, eviden^^e 
suggests the same link for principals. 

As suggested by the availability of documents, the specifications in the policy tended to 
be most thorough for teachers and least thorough for superintendents and directors In 
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practice, distinctions between administrative and developmental appraisals, formative and 
summative appraisals, and the (ike were not common; one policy tended to be used for all 
purposes. Exceptions occurred in the case of teachers on probationary contracts and, in so 
boards, for teachers who had been placed under review. 



Impact of Policy and Practice 



Moderately high levels of compliance were reported for all appraisal policies; typically 
about one-quarter believed policy was followed exactly and another one-half that practice was 
approxim-Uely as called for in policy. Significant portions of the remaining respondents were 
not sure. At the same time, compliance ried considerably among boards, with those reporting 
the highest levels of compliance for their teacher and principal policies also reporting the 
most effective systems. 

Most effort has been committed to implementing and carrying out teacher and principal 
evaluations, and least effort has been made ii superintendent ar*d director appraisals. The 
degree of effort is reflected in resources committed, documents prepared, workshops held, 
skill of the evaluators, and how seriously th? whole process is taken. In the case of 
teachers and principals, boards appear to have made a modest effort to implement their 
policies while the individuals involved have made a moderate effort in conducting the 
appraisals themselves. For superintendent evaluations, it appears that, on average, only 
minimal efforts have been made on both (ounts, and there is a noticeable dismay among 
superintendents concerning this situation. For directors, trustees seem serious but lack the 
skills necessary to do an adequate job; this may reflect a lack of effort or a lack of 
knowledge on the part of trustees concerning opportunities to improve the quality of the 
appraisals they conduct. 

The majority of evaluatees of all ranks reported some improvement in their performance as 
a result of their appraisal; figures ranged from 60 per cent for teachers to over 80 per cent 
for principals. However, in all cases most of the respondents indicated that there had been 
only a slight improvement. Those reporting a substantial improvement ranged from a low of 3 
per cent for teachers to a high of 9 per cent for superintendents (in spite of the low marks 
superintendents gave their appraisal process). 
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Those in higher positions tended to perceive more improvement on the part of their 
subord-^nates than did the subordinates themselves. For example, superintendents believed 43 
per cent of the teachers showed substantial improvement. This pattern holds all the way to, 
but not including, the director's position. Whereas 44 per cent of the directors reported 
that they perceived modest improvement in their own performance, only 23 per cent of the 
trustees reported seeing this improvement. No directors or trustees reported substantial 
improvement in a director's performance. The conservative views of trustees carried over to 
other evaluation programs: on a 5-point scale measuring how successfully the evaluation 
systems for each group were achieving their goals, their responses averaged between 3.0 and 
3.2, i.e., moderate success. 

On other measures, such as perceived fairness of the process, skill of the appraisers, 
and one'*^ attitudes after the appraisal, results were quite positive. Eighty per cent of each 
group indicated that the judgement of their appraisers and the nature of the evaluation 
process were fair. In fact, a majority of all groups except directors reported feeling good 
as a result of the process. While evaluation is often portrayed as a process to get people to 
"shape-up or ship-out", it appears that it has more the effect of a coach's pep-talk before a 
team is sent out to play. In fact, boards that were forced to dismiss teachers were tho^o 
that displayed the least effective, not the most effecti ^, appraisal systems. 

Demographic factors may also play a role in explaining the effectiveness of an evaluation 
system. Suggestions have been made that the size of a school board, its rate of decline in 
enrolment, the age of its staff, and the like may colour the process. Analysis of the 
relationship of a number of demographic variables to the effectiveness of evaluation suggests 
that there is a set of variables associated with secondary education (percentage male, 
percentage with degrees, and so forth) that are linked to less effective evaluation. Whether 
the system of evaluation or the nat.ure of the evaluatees accounts for this phenomenon cannot 
be answered with the data at hand, but it would seem logical that more experienced and more 
highly educated individuals would be less likely to profit from formal appraisa . 

The survey results confirm the picture portrayed in the preceding chapcer, i.e., Ontario 
school boards have been making an increasing commitment to the performance appraisal of 
educational staff at all levels. Considerable progress has been made, but considerable room 
for additional progress is available. Teacher evaluation systems are most prevalent though 



apparently lacking effectiveness; principal systems are less prevalent but appear to be 
more effective; and superintendent systems are still less conmon and apparently are 
considered inadequate, though not necessarily ineffective. Appraising the director's work 
presents a special problem there is no formal organization to rely upon and much of the 
initiative must come from the director. 

Tn this chapter, the performance appraisal practices in a large sample of Ontario boards 
have been described; however, links have not been drawn between the specific policies of a 
given board, the processes that that board has used to make and implement decisions, and the 
outcomes of this p'^ocess. These matters are considerad in the following chapter, which 
provides a cross-case analysis of performance appraisal in eight Cntario school boards. 
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CHAPTER 4 

CASE STUDIES OF PERFORMANCE APPRAISAL PRACTICES: A CROSS-CASE ANALYSIS 



Purpose of Case Studies 

Analysis of survey data alone (chapter 3; provided a representative estimate of the perceived 
status of performance appraisal of those in all roles of interest. It indicated, for the 
province as a whole, the opinions of appraisees and appraisers about how appraisal was carried 
out, what types of activities led up to and followed it, and what wer« the perceived strengths 
and weaknesses of the procedures used. The focus of that analysis was on roles: teachers, 
principals, superintendents, and Directors. Analysis of school board policies (chapter 2) 
provided a picture of the level of policy development in the province and the alternatives 
advocated for carrying out specific components of the appraisal act. By comparing the results 
of policy analysis with the results of the survey, we documented the similarity of policy to 
perceived practice. 

Survey data and policy analysis data, however, are not particularly sensitive to the 
social/organizational context in which appraisal is conducted. Such data tend to divorce the 
act of appraisal from the situation in which it is conducted. Yet the situation contributes a 
good deal to the meaning that both appraiser and appraisee attach to the appraisal act. For 
example, the same procedure for appraising principals has very different meaning (impact) in a 
board where demotion and promotion decisions ha^e been based, historically, on such appraisal 
than in a board where formal appraisal data and promotion decisions are uncoupled. One 
important purpose for conducting case studies of appraisal practices in eight school boards, 
then, was to recognize the role of context and permit its effects to su^^^ace. The second 
purpose was to explore, in more depth, many of the same issues addressed by the survey data. 
In particular, the case studies examined the perceived impact of appraisal systems and factors 
associated with such impact. 

Three sets of questions were addressed in the case studies. First, what a^^e the actual 
appraisal practices of these school boards? Specifically, what procedures are used to prepare 
for appraisa,, to collect and analyse data, and to report and act on tne data? How do these 
procedures differ within case boards across roles? How do procedures differ within roles 
across case boards? 
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The second set of questions was concerned with how appraisal practices come into being 
and evolve over time. Specifically, who participates in the development of these policies? 
Of what does this process consist? What steps are taken to implement the policies? 

A final set of questions focussed on various manifestations of the impact of appraisal 
practices. How much effort is devoted to appraisal of those in different roles? To what 
extent are practices consistent with policies? How do those be.ng appraised perceive the 
impact of such appraisal on their own performance? 

These three sets of questions and the methods used to answer them are consistent with the 
type of research advocated by Knapp (1982) on the basis of his review of the state of the art 
of teacher evaluation research. 

Method 

Design 

Case study designs have long been advocated as useful in helping to develop hypotheses, to 
illustrate general principle^, and to better understand relationships among variables in 
specific but "whole" contexts. Their potentid internal validity has been acknowledged, their 
external validity assumed to be low. But recent interest in qualitative re<^earch strategies 
has included re-examination of these assumptions and further refinement of case study designs. 
In this study, attempts were madp to reflect developments in case study design intended to 
increase their external validity (e.g., Kennedy, 1982; Miles & Hubcrman, 1984). This attempt 
included the collection of data in each case using a common framework established prior to 
data collection and the codification of data from each case in common categories for analysis 
and reporting. 

Techniques used for data collection were convergent (Campbell and Fiske, 1959). They 
accumulated data from different sources using different instruments abotit many of the oame 
questions; questionnaires, interviews, and document analyses were employed in arr-'ving at 
answers U a cluster- of common questions. As well, each source of data was designed to answer 
some unique questions. 
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Recent attention has been given to the value of integrating fieldwork and survey methods 
(e.g., Louis, 1982; Miles, 1982; Siebsr, 1973). In the present study, data collected using 
fieldwork methods (individual and group interviews, and document analysis) were considered to 
strengthen the survey results by (1) adding to the validity or "persuasive plausibility" of 
the survey results; (2) helping interpret statistical relationships evident in survey results; 
(3) helping illustrate apparent, prototypical appraisal practices; and (4) clarifying puzzling 
responses to survey questions. 



Survey results helped to strengthen fieldwork (interview) data by (1) providing a basis 
for sampling case beards to be studied that would ensure variation in approaches to 
performance appraisal (rather than just variation in demographic characteristics, for 
example); (2) helping to demonstrate the generality of findings from single cases; (3) helping 
to verify the generality of findings common across case studies; and (4) casting new light on 
case study data. 



Sample 



The survey sample was described in chapter 3, and the sample of documents reviewed were 
outlined in chapter 1. Using data from these two sources, we selected eight boards which 
appeared to vary in level of appraisal policy development, sophistication of appraisal 
practices, and apparent importance attached to performance appraisal. The eight boards were 
also quite diverse in size and geographical location in the province, '^'^levant features 
of each board are described in the separate case reports fur each (Volume 3, Appendix C). 

Table 93 summarizes the number of people within each role in each of the eight boards 
that provided interview and questionnaire data. Note that consultants were not included in 
the questionnaire survey, and that no trustees were interviewed as part of the case studies. 
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Table 93: Sample in Case Boards Respon d ing to Interviews/Questionnaires 



Sampling Unit/ BOARD 



Role 


A 


B 


C 


D 


E 


F 


G 


H 


School 

Elementary 
Secondary 


4 
2 


8 
4 


1 1 
4 


15 
8 


10 

5 


13 

0 


10 

0 


/ 

0 


Direc to r 
Interviews 
Quest . 


1 
1 


1 
1 


1 

1 


1 

0 


1 
1 


1 

0 


1 
1 


1 

1 


Superintpndent 
Interviews 2 
Quest. 2 


4 
4 


5 
4 


4 

10 


4 

12 


5 
9 


3 

5 


3 
3 


Principal 
Interviews 
Quest . 


4 
12 


4 

35 


3 

50 


5 
95 


4 

58 


6 
29 


4 
41 


4 
22 


Teacher 
Interviews 
Quest. 


8 
17 


7 

163 


8 
261 


15 
380 


8 
227 


12 

208 


8 
92 


7 
56 


Consultant 
Interviews 
Quest . 


2 

0 


0 
0 


2 

0 


3 

0 


0 

0 


0 
0 


2 

0 


2 

0 


Other 

Interviews 
Quest . 


0 
0 


0 
0 


0 
0 


0 
0 


0 
0 


0 
0 


2 

0 


0 
0 


Groups^ 


F 

P(2) 


C 
P 
i. 


c 

F 
F 
T 


c 
p 

Res. 


F 

m 
i. 


c 

T 


c 

P(2) 
T(2) 
Tr(2) 




Trustees 
Interviews 
Quest . 


0 
3 


0 
2 


0 
2 


0 
0 


0 
2 


0 
2 


0 
5 


■z 



Applies to questionnaire sample only. 
Not included in the survey study. 
^ Applies to interview sample only. Key: C=-Consultants ; F= Federation; 
P=Principals; Res. T^Resource Teachers; T=Teachers; Tr=Trustees. 
Numbers refer to the number of groups. 
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Instruments 



Instruments developed to collect individual and group interview data, incorporated all classes 
of question raised by the survey questionnaires. In addition, questions concerning impact on 
specific areas of performance were developed (see Volume 3, Appendix C-I). 



Analysis 



Data from individual case boards were analysed separately by those who collected the data. A 
common framework was used to guide the analysis. The cross-case analysis described here was 
prepared by one member of the research team using the individual case reports, as well as the 
original survey data and board documents. This member of the research team was not involved 
in the actual writing of any individual case reports. 



Results 



Actual Appraisal Practices 

In this section, a comparative description of the case study data is presented within a 
framework of performance appraisal system components and roles. 

Case study data concerning each category of appraisal system characteristics for the 
roles of director, superintendent, principal, teacher, and consultant are presented in tables 
94, 95, and 96. Each of these tables identifies the range of variation in apprai-al practices 
and indicates the variant{s) adopted for each role in each board. These variations usually 
represent independent, alternative practices. However, in several cases a board was 
identified with more than one alternative as a way of keeping manageable the number of 
alternatives displayed in the tables. Tables 94, 95, and 96 also indicate when no data were 
available (NO) or where nothing was being done with respect to that component of the appraisal 
(NA); placement of a board in this category was mostly for the latter reason. 



ERIC 



27 { I 

255 



Preparation 



Planning 

The close involvement in and control over their own appraisal by directors appeared to have 
reduced the necessity for much advanced planning for appraisal activities.^ Considerably more 
planning was characteristic of appraisal practices with all other roles. Such planning 
appeared to be most extensive with principals; in all but one board they had an opportunity to 
influence the nature of the appraisal, usually determining the goals t>at would serve as the 
criteria for evaluation by participating in goal-setting. Three boards provided the same type 
of opportunity to teachers (C, D, H) and two boards to consultants D). Only in board A, 
where the policy was still being developed, did thpre appear lo be little or no advanced 
planning for teacher appraisal; in this case, the nature and extent of the planning depended 
on the individual principal. 



In case board E, the appraisal of the director was particularly difficult to report. The 
written policy had been prepared by a director who was on leave during the period of our 
study. The atc.ng director who responded to our questionnaire and interview had no 
intention of following the policy established by his predecessor. 
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PAS Components 


Dir 


Supt 


Prin 


Teach 


Cons 


1 . 


Planning for appraisal: 












1 .1 


opportunity to understand, 














influence 


u 
n 




DLDt A 




uR 


1 .2 


effective communication 














about procedure 




n 
U 




•nr» D 


GC 


1 .3 


communication about 














procedure 








r 


1? 

r 


1 .4 


little or no advanced 














prepa ration 








A 
n 




1.5 


no data/not appropriate 


BEACD AF 


u 
n 




DP Cjil 






GE 










2. 


Purposes for appraisal 














(goals) : 












2.1 


balance education, adram.. 














policy i?oals 


BHGE 


DBGE 


Disr U A 

n 






2.2 


emphasize education goals 




A 






A 

A 


2.3 


emphasize admin, goals 


A 






A 


D 


2.4 


emphasiz3 policy goals 






rr 
L 






2.5 


no data/not appropriate 


FCD 


CFH 


H 




BFGEH 


3. 


Criteria for appraisal: 












3.1 


job description 


u 
li 




A 


DCAB 


Ar 


3.2 


objectives : 














collaboratively set 


FG 


DCAB 






r u 
K> n 








GE 








3.3 


objectives : 














set by appraisee alone 


CB 


H 


DC 






3.4 


mixed criteri a/ uneven 














collaboration 


E 


BE 


nur 


Ur 


Dt J? 


3.5 


no data/not appropriate 


AD 


F 






BE 


4. 


Standards for appraisal: 












4.1 


explicit, in policy 


H 




n 

\j 


n 




4.2 


explicit, set by 














appraiser 






A 


n A 
1/ A 




4.3 


explicit, set 














collaboratively 




BE 


BFC 


CBG 


CG 


4.4 


implicit, set by 














appraiser 


ABE 


CAGH 
D 


ur u Liii 


ur uii 


i/T n 




no data/not appropriate 


FCDG 


F 






ABE 
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Purposes 



The questionnaires and interviews used in the study allowed many specific appraisal purposes 
to be identified. These have boen classified as educational (e.g., improved student 
learning), administrative (e.g., pri'^:ipal transfers), ard policy-oriented (e.g., implementing 
Bill 82) purposes. Boards B, E, G, and H appeared to address all three categories of purpose 
through tiieir director appraisal. Board A was without a formal director appraisal procedure; 
nevertheless, it used an informal procedure in the year preceding the study to serve the 
administrative purpose of determining the size of the direc^^or's salary increase. 

Considerable variation among boards in the categories of purpose to be served by 
appraisal is c;ident in relation to superintendents, teachers, and consultants. Boards B, D, 

and G pursued all three categories with superintendent appraisal; boa d A emphasized 
educational purposes. Boards C, D, F, and G pursued all three sets of purpose with teacher 
appraisal; boards A, B, and H emphasized educational purposes while board A, with its policy 
Sv'ill being developed, seemed to be pursuing both educational and administrative purposes. 
Educational and administrative purposes were served by consultant appraisal in boards A and D 
respectively. 

All three categories of purpose were pursued by six case boards th^^nugh p; incipal 
appraisal. Board E emphasized policy goals. 



Criteria 



Job description? and objectives were the two criteria usea •'"n appraisal practices for all 
roles. Objectives were sometimes set by the appraisee alon^^ ^nd sometimes in collaboration 
with the evaluator. In no case were objectives set by the evaluator alane. 



Job descriptions alone were used more prcminer.+ ly with those roles lower in the 
organizational hierarcny (board E is an exception); four boards used such descriptions as 
criteria for appraising teachers, two boards for appraising consultants, and one board for 
appraising princ'jaU. Two boards combined oojectives anc job descriptions in the appraisal 
of teachers (D and F) and three boards did this for consultant: (C, D, and F). Boards 0, F, 
and H used such a combination with principals. 



Three boards employed a job debcription in the appraisal of senior staff (B, E, and H) 
and these boards used collaboratively set objectives as well The appraisal of 
superintendents and directors was largely conducted around collaboratively set oojectives. 



Standards 



Standards are the levels of performance within classes of behaviour or criteria used as the 
basis for judging the adequacy of an appraisee's performance. The standards used for 
appraisal varied in terms of how they were set in policy, by the appraiser, by the 
appraisee and whether or not they were explicitly identified. Considerable variation is 
evident in practices both within and among the case boards. Three boards used implicit, 
appraiser-set standards in assessing the director (A, B, and E); five boards used this 
practice with superintendents (A, C, D, G, and H). In boards B and E, standards were 
collaboratively and explicitly set for the appraisal of superintendents. 

Case boards, as a whole, used all variations for setting standards in appraising 
principals and teachers. In three cases these standards were made explicit, but boards 'i, E, 

G, and H deviated from this practice for principals, boards D, E, F, and H for teachers, 
and boards D, F, and H for consultants. 

Data Col lection 

Soirees of Information 

Table 95 presents information about how data were collected and analysed in the appraisal of 
all roles. When a single source of information was relied on exclusively, it was always the 
appraisee; this was the practice in board C for consultants, in boards A, C, and E for 
teachers, in boards A and E tor principals, and in boards t and H for the director and 
superintendents, respectively. Multiple (more than three) sources of information, potentially 
the most expensive but reliable alternative, were adopted consistently only by cwc boai ds (B 
and D), although they were alco used by boards G and H in principal appraisal. Two or three 
sources of information, including the appraisee, were used by boards A, B, G, and H for the 
director, boards A, C, E, and G for superintendents, boards C and F inr principals, boards F 
and G for teachers, and boards A, F, and H for consultants. 
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Table 95: Data Con^^ction for Appraisal According to System Components and Staff Role in 
Each Case Board 



PAS Coaponents Dir Supt Prin Teach Cons 



1* Sources of information: 

1.1 multiple, including 
appraisee 

1.2 several, including 
appraisee 

1 ,^ appraisee only 
1.4 no data/not known 



2. Types of information. 

2.1 multiple 

2.2 two or three 

2.3 primarily one 

2.4 not clear 

2.5 no data/not known 



'5* Collectors of information: 

3»1 an individual 

3»2 two or three people 
^•3 a team 

3»4 no data/not known 



4. Time spent collecting 
info rmation: 



4.1 


a week or more 


3 


DBG 




A 


4.2 


two to four days 


H G 


EH 


G 




40 


a day 


D 


CA 


DFCE 




4.4 


less than a day 


E CEHA 


F 


A3 


c 


4.5 


no data/not known 


ABFCD F 
G 






BGEH 



BE 



:)BCH DB 



ABGH CAGE CF FG AHF 
E H AE CkE C 
FCD F BGE 



H 


B 


DBFAH 


DG 






DCG 


G 


CFE 


D 


BE 


H 


E 


ABH 


CA 


AG 


AF 


C 




F 


FCD 


E 






BGEH 



ADCF CFGH DABFC ACG 
EGH EGHH 
ABEG B BAE 
H D D 

FCD BEH 
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Types of Information 



Independent of the source, there are sevcTdl different types of information that may be 
collected. Self-report information using interview or questionnaire instruments, direct 
observation of behaviour, and written records (lesson plans, daybooks, memoranda to parents, 
and the l^'ke) are the most frequently collected types of information gathered during 
appraisals. Most case boards relied on two or three such types for the appraisal of most 
roles. Multiple types of information were collected most frequently in the appraisal of 
principals (boards A, B, D, F, and H). One or several types were used by most boards for most 
other roles. Observation data were usually collected only in the appraisal of teachers. 

Collectors of Information 

Information was most frequently collected by one oe'^son in appraising most roles. Boards A, 
B, and E deviated most from this practice in us.ng two or three people for selected roles. 
Board D used a team of people for both principals and consultants. A group of trustees in 
board H also collected information on the director. 

Time Spent 

Collecting information to appraise a single individual was often not a time-consuming activity 
for appraisers in the case boards; normally it took a day or less. This relatively modest 
amount of time frequently became unmanageable, however, when multiplied by the total number of 
individuals to be appraised over the period of a year. In spite of this cost, board B 
appraisers appeared to spend at least a week collecting information on superintendents and 
principals although ..oncerns were expressed about this ^ime in relation to principal 
appraisal. This was also the practice in boards D and G with principals and board A with 
consultants. Extensive expenditures of time were often accomp'T'.eH by actices that 
permitted the time to be distributed over an extended calendar period. But this was not 
always so, as illustrated in the case of principal apprai.a! in board D, ^or example. 
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Reporting and FoUow-Up 



Nature of Report 



Information about how appraisal data were reported and the nature of actions surrounding such 
reports are presented in table 96. Boards A, E, and G relied on verbal reports to the 
director, A to all but teachers, C only to superintendents, G to superintendents and 
consultants, and H to consultants only. Relatively brief written reports were prepared by 
three of the remaining boards for principals (B, F, and H) and teachers (A, E, and F). 
Detailed, written reports were prepared for all but the principal's role (and possibly 
consultant) by B diJ all but the director's role by D. Board H prepared such reports for the 
director and teacher's and G for principals anci teachers. 



Table 96: Reporting and Follow^Up for Appraisal According to System Components and Staff 
Role in Each Case Board 



PAS 


Components 


Dir 


Supt 


Prin 


Teach 


Cons 


1 . 


Nature of Report: 












1 .1 


written, detailed 


BH 


DB 


DE 


DCBGH 


DC 


1 .2 


written, brief 




EH 


FBH 


AFE 




1.3 


verbal, detailed 




A 






A 


1 .4 


verbal, brief 


AGE 


CG 


A 




OH 


1 O 


no uc«ta/not known 


FCD 








BE 



2. Destination of report: 



2.1 not kept AE C AGH 

2.2 giver to apprp.isee only G A 

2.3 filed in board office G DBE DFB DC3F DCF 

2.4 no data/not known BFCDH AH GEH AGEH aZ 

Report sharing: 



3.1 discussed, possibly 
revised 

3.2 presented and e>-plained 

3.3 sent written report for 
information 

3.4 not shared 

3.5 no data/not known 



BFGE DBGH DEFAGH DCBGH CAGH 
Ah r AFE 

C F D 

C 

CD AE BE 



4. Follow-up; 



4.1 no explicit follow- up 

4.2 plan developed, 
not monitored 

4.3 plan developed, monitored 

4.4 no data/not known 



AGE ACFH AH ADCFBE CG 

G CF D 

DB DBGE A 

BFCDH GH BEH 
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Destination of Report 



The dominant practice among the cases studied was fo'^ the report, after discussion with the 
appraisee, to be filed in the board office. Board A deviated most from this practice; three 
boards (A, G, and H) did not keep reports of con'ultants' appraisals. 

Report Sharing 

Most boards shared reports witn the person being appraised, usually offering the possibility 
of revision of the report if inaccuracies were identified. Board C deviated most from this 
practice, although it was not alone. Discussions did take place between director and 
superintendent > ^ not specifically focussed on a report. This may have resulted from the 
still informal status of policy governing appraisal practices for superintendents in C at the 
time of the study. In boards C, D, and F a written report was sent to principals, teachers, 
and consultants, respectively, although in C principals could respond in writing to the 
report, if they chose. And in board D, the evaluation team met with the principals, who also 
were encouraged tc share the results with school staff. Boards A and H presented and 
explained the report to the director, E to principals, and A, E, and F to teachers. 

Fol low-Up 

Of the three types of follow-up practice evident in the case boards, "no explicit follow-up" 
was the most frequent practice. This was so in boards A, B, C, D, E, and F with teache'-s. 
Boards C and F developed follow-up plans which were not monitored with principals (in D, 
foi:ow-up was usually the practice only when a problem was detected); D and G also followed 
this practice with consultants and superintendents, respectively. Boards B and D developed 
and systematically monitored progress with follcw-up p^ans in the case of superintendents and 
principals; E and G also followed this practice with principals, as did A with consultants. 
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Evolution of Policy 



Implementation 

Table 9/ summarizes tht nature of implementation of activities in the appraisal of ail roles 
in each of the eight boards. None of the boards reported explicitly addressing the 
implementation of their director appraisal practices. In most of the boards, the directois 
had been closely involved in developing the procedures, in seme instances inHiating such 
development. This obviated the necessity of formally attending to implementation Only 
boards B and H reported attending to this issue for superintendents. Implementation of 
principal policy, however, involved at least the formal provision of information about the 
policy to principals (A, F, and G). except in H. In three cases (C, D, and E), limited 
training a day or two of ^'n-service was provided. One board (B) provided substantial 
training in the form of "very thorough" workshops, although the intensity of the training 
appeared to vary across regions in the board depending on the initiative of the area 
superintendent. 



T^l e 97: Evolut i o n of Appraisal Policy and Practice According to System Components and 
Staff Role in Each' Case Board 



PAS Components 



Dir 



Supt Prin Teach Cons 



1 . Nature of implementation 
process 

1.1 not addressed explicitly 

1.2 provision of information 
about procedure 

1.3 limited training in use 
of procedure 

1.4 substantial train?.ng 

1 .5 no data/not appropriate 



ABDFGE DCAF H 
EG 

H FGA 

LCE 
B B 

CH 



2. Degree of policy specification; 

2.1 no pol my/ implicit, 

poorly unders*:ood CD 

2.2 implicit but well 
understood AH 

2.3 explicit, briefly 
specified FE 

2.4 explicit, extensively 
specified BG 



CH 
AEH 



CFE 

DB 

H 



DBFG DCBFG DCBF 
E EHG 



CAB 



BHG 
A 



DCB 
E 
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Board A had not yet explicitly addressed the implementation of their teacher practires. 
This may have been because of che newness of the policy and its tentative status at the time 
of the study. All remaining boards appeared to provide information (G), limited training (C, 
E, and F), or substantial training (B,D) to teachers at least in the early stages of policy 
implementation. Substantidi training in implementing policies was provided to consultants by 
only board D; th° remainder did not explicitly address this problem. 

Degree of Specification 

Most of the case boards had explicit, extensively specified policies for the appraisol of all 
but the director. Only boards B anc : had such a director policy; however, B, G, and H had ) 
consultant policy. Brief bui. explicit policies were provided by boards E and F for directors 
and A, E, and H for superintendents. Boards A and H had implicit but well understood policy 
for directors, A for consultants and C and H for superintendents; H had similar policies for 
all but consultants. No specific oolicy, implicit or explicit, was available in C or D for 
director appraisal or in A for principal and teacher appraisal. 

Impact of Policy and Practice 

Each of the three sets of variables clustered together in table 98 concerns the impact of 
appraisal practices. These variables range, however, in the directness with which they are 
estimates of actual change in performance as a consequence of t^praisal practices. The set of 
variables that comes closest to such an estimate is the perceived "nature and degree of 
impact". "Degree of compliance with policy" is likely the least direct estiirate of impact; 
high levels of compliance with an ineffective policy, for example, are not likely to influence 
performance much. "Extent of appraisal effort" migh, be considered a mediating variable 
plausibly exercising a strong influence on tht degree of impact on perfo'-mance. 
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Table 98: Impact of Appraisal Policy and Practice According to System Components and Staff 
R ole in Each Case Board 

Outcomes Dir Supt Prm Teach Cons 



1 . Degree of compliance with 
policy: 



1.1 


low level of compliance 










1 ,2 


low, compliance m 












specific areas 




E 




G 


1.3 


high, deviance m 












specific areas 


G 


LG 


ACBFGE 


FA EH ADC 


1.4 


high level of compliance 


ABH 


CABH 


D 


DCB 


1.5 


no data/not appropriate 


FCDE 


F 


H 


BEH 


2. 


Extent of appraisal effort: 










2 . 1 


performance not appraised 


C 


F 




BE 


2.2 


sporadic appraisal 


E 


E 


A 


H 


2.3 


regular , infrequent 












appraisal 


F 




DF 


DCABF D 


2.4 


regular, frequent 












appraisal 


ABGH 


DCAB 


CBGE 


E CAG 






GH 


H 




3. 


Nature and degree of 












impact: 










3.1 


negative impact on 












performance 






A 




3.2 


no perceived impact on 












performance 


AFCE 


FE 


C 


CBE B 


3.3 


mildly positive impact 




DC AG 


DFEH 


DAF DCG 


3.4 


significant positive 












impact 


BGH 


^ 


BG 


H A 


3.5 


no data/not appropriate 


P 


H 




EH 
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Degree of Compliance 



The case boards demonstrated relatively high levels of compliance in implementing existing 
policies as applied to most roles. Where compliance was generally high but some deviance was 
found, tho deviance could usually be traced to such time-related matters as less frequent 
appraisals than specified, insufficient follow-up, and too little data collection. Inadequate 
skill on the part of the evaluator and policies still under development were also causes of 
deviance from policy. Significant disagresment with the nature of the policy did not appear 
as an explanation of deviance from practices set out in policy. Boards E and H are exceptions 
in this respect: in E, particularly, policies for administrator appraisal are widely 
criticized; in H, old and new policies overlap in practice and have yet to be effectively 
combined. 



Extent of Effort 



Regular, either frequent or inf.-equent, appraisal activity was characteristic of most case 
boards for most roles. At least annually, A, B, G, and H appraised the director. A, B, C, D, 
G, and H superintendents, B, C, E, G, and H principals, and A, C, and G consultants. 



Teachers in all boards were appraised re^jlarly but frequencies ranged from once every 
three to once every five years. Board E was the only one which appeared to appraise teachers 
annual ly. 



Nature and Degree of Impact 



Negative impact on performance was reported only by principals in board A. Mild, positive 
impact describes the effect on oerformance most frequently reported by all roles in all 
boards. The director in B, G, and H, the superintendents in B, the principals ir B and G, and 
the consultants in A reported ir.ificant positive impact. No impact on perfomance was 
reported by the director in A, C, E, and F, superintendents in E and F, principals in C, 
teachers in b, C, and E, and consultants in B. 




Summary and Conclusion 



Considerable variation was evident in practices associated with preparation for performance 
appraisal in the case study boards. This was true for all roles, even for the teacher, where 
recent legal disputes in the province about the status of teacher appraisal practices leading 
to dismissal might create the expectacioii of uniform care among all boards. Preparation was 
most systematic overall with principals and least systematic with directors. 

Planning immediately prior to appraisal was standard practice for most roles. Purposes 
for appraisal were often balanced amuog educational, administrative, and (to a lessee extent) 
policy goals with little variation due to role. C 'teria used in appraisal were usi-ally based 
on either a job description, a set of objectives, or both. Standards tended to be left 
implicit or were viewed as something to be negotiated between appraisee and appraiser. Rarely 
were they made explicit in policy. 

Procedures for data collection and analysis appeared to vary widely across boards and 
roles. Some consistency within boards was evident, however. This consistency was most 
evident across administrative roles considered as a group; data collection and analysis for 
teacher appraisal were usually Quite different than that for administrator appraisal- Two 
factors seemed to account for this contrast: the larger number of teachers as compared with 
administrators and the ease ith which classroom instruction, in contrast vith admin . otrative 
behaviour, lends itself to direct observation. More effort for each appraisee was devoted to 
administrator (especially principal) than teacher appraisal. This is the most obvious, single 
explanation of differences in the reported impact of appraisal practices on performance. 
Although the time formally devoted to superintendent appraisal tended to be modest and impact 
on performance comparatively high, many informa' opportunities for appraisal were usually 
available to the appraisee. 

For the most part, with all roles, appraisers prepared and discussed with appraisees some 
form of written report. It was minority practice, however, to develop a pian for following up 
recommendations from the report and rare, indeed, for such plans to be monitored in any 
systematic fashion, except in the case of principals where the practice was much -ore common. 
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Among the eight boards, actions consciously designed to implement appraisal policies were 
most common for teachers and principals. Policies were also specified in greatest detail for 
teachers and principals and least for directors and consultants. 

Estimates of the impact of appraisal activities suggested relatively high degrees of 
compliance with policy, and activities that are regular but varying substantially in their 
frequency; teachers were normally appraised every three to five years wheieas annual appraisal 
was more common practice with most administrative roles Impact on perfc ^'-nce was usually 
perceived as mildly positive for all rcles; nevertheless some boards, especially T appeared 
to be using a system with high impact, at least in relation to administrative performance. 

These case study data warrant the following conclusions 

1. The greatest overall "success" in developing and irnj- lamenting performance appraisal 
practices perceived as effective was in .cation tz school and school system 
admi ni strators. 

2. Appraisal practices for such admini st) ator:, 'particularly principals; tended to be 
characterized by careful preparation for appraisal, negotiated criteria and standards, 
substantial amounts of data collection, and regular, relatively frequent appraisal. 

3. The major difficulty associated with effective teacner appraisal was the burden of the 
task for administrator? , given the numbers of teachers to be appraised in the period of a 
year. The size of the burden seemed to substantially accojint for tendencies towards 
perfunctory preparation, use of standardized criteria, reliance on limited amounts of 
data, and relatively infrequent appraisal, 

4. The most apparent deficiencies among appraisal practices, as a whole, in the case study 
boards were: (a) the use of criteria not explicitly linked to role effectiveness; 

(b) the non-specific and perhaps irrelevant feedback provided as a consequence of some 
appraisals; (c) the tendency to limit follow-up primarily to staff with diagnosed 
problems, a practice quite unsatisfactory to the large majority of remaining staff; (d) 
the limited attention given to planning for follow-up and, even more glaring, the neglect 
of monitoring plans for follow-up. 
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5. Th<»re was consideraDle variation in appraisal practices among case boards for all roles 
Among these boards could be found instances of practices that likely deserve the label 
exemplary. Careful scrutiny of these exemplary cases would be helpful for many boards 
interested in improving their practices. 
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CHAPTER 5 
CONCLUSIONS AND IMPLICATIONS 



In this chapter, the findings from the policy analysis, questionnaire survey, and case studies 
are integrated with the framework developed in chapter 1. From this integration comes a 
number of implications concerning the use of performance appraisal in Ontario school boards. 
This chapter does not include a summary of the findings reported in the various phases of the 
study, for this, the reader is referred to the overview of the study provided in chapter 1 
and the summaries found at the ends of chapters 2 through 4. 

Figure 1, shown on page 3 of chapter 1, depicts a systematic view of the position of 
performance appraisal within an organization. The left side of the figure shows the 
organizational structure, roles, and so forth, while the right side outlines the various 
stages of the appraisal process. Linkages between the two are suggested, but these linkages 
may or may not exist. It is the existence and nature of these linkages that are of central 
concern in this chapter, in which a stance is taken that linkages ought to be present. If 
they are not. and performance appraisal operates independently of the fundamental structure 
and operations of the organization, then appraisal is at best a hollow ritual. However, if 
links are developed, these links, if exploited appropriately, can ensure that information 
developed during the appraisal process can be used to make the organization more effective by 
changing, as appropriate, its goals, structures, and operational processes. 

Preparation 

The first question, then, is the extent to which the organizational goals and objectives, 
organizational structure, job descriptions, and employment contracts are linked to preparation 
for appiaisals of directors, superintendents, principals, and teachers. 

For directors, ano to a large extent for superintendents, there was a clear lack of 
direction from most school boards, and from public boards in particular. Senior professional 
educators voiced a strong desire for the purposes of their appraisal to be linked to 
organizational goals and objectives, but in many cases they found it necessary to develop 
those goals themselves and to request concurrence from the trustees. 
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In addition to problems related to organizational goals, a structural problem also arose 
in carrying out director appraisals in that directors were often forced to initiate the 
process themselves. As well, the actual execution of the appraisal often revealed a lack of 
skill on the part of trustees: directors did not find the experience one that helped their 
morale, as did those in other roles whose evaluators were professional educators. This 
finding suggests that depending solely on trustees to carry out director appraisals is 
probably unwise and that another structure, perhaps involving a professional organization, 
professional consultants, or the Ministry of Education as well as trustees, is needed. 

Since objectives-based appraisals were most frequently used with all administrators, the 
relationship of annual objecti'^s to their job description and the criteria used in their 
appraisal is of interest. In a sense, a job description provides a list of permanent 
objectives to be achieved, while the objectives set for a particular year and the criteria 
used to assess the achievement of these objectives are the transitory aspects of the job. 

Surprisingly, there was little evidence that this type of distinction was made in 
practice; it was not unusual to see annual "objectives" that were permanent elements of a 
job, with no reference being made to the job description. In other cases, there was total 
reliance on objectives-based appraisals with no attention being paid to the perennial aspects 
of a job. This problem was most evident at the superintendent level; many superintendents 
lacked feedback on their overall job performance. 

Employment contracts apparently had more effect on the purposes of appraisal for 
directors, and to some extent super-' ntendents, than they did on the criteria. For directors 
on term contracts or those whose contracts had merit pay clauses, renewal of the contract or 
the granting of increments was tied to their level of performance as judged ^n formal 
performance reviews. As well, directors and superintendents often admitted that having good 
appraisal reports on file offered them job security. If their contracts were terminated, they 
believed that they could call upon these reports as evidence that the termination was not 
justified. Further, if their dismissal was sustained, they believed good appraisals could 
serve as bargaining chips ensuring a more nenerous settlement. 

Where contracts with directors and superintendents did mention criteria, che criteria 
were usually those associated with achieving objectives. Hence, the criteria would change 

288 

273 



annually or biannually as new objectives were set. Similarly, pre-conferencing for directors 
and superintendents was normally tied in with the process of setting objectives. 



For principals the general picture of linkages between the organizational sector and 
preparation for performance appraisal was more positive than was the case for directors and 
superintendents. The widespread use of goal packages for schools would seem to assure ties to 
the goals and objectives of the school board and Ontario government. In most boards, the 
shift in the role of superintendent from evaluating teachers to evaluating schools and 
principals is complete, so the structural ambiguity of the relationship of the superintendency 
to the principal and school has been resolved. In only a few cases were the two vying for 
responsibility for the evaluation of teachers. As well, the developmental purposes of 
principal appraisal are unchallenged and detailed lists of either criterion domains or 
specific criteria, while not always tied formally to job descriptions, usually served to 
ensure broad-based appraisals, not appraisals focussed narrowly on specific goals. One 
problem does remain in this area, however, and that is the role of the principal in curriculum 
implementation and program evaluation. It appears that, at present, performance appraisal is 
carrying the burden of realigning the principal's role to emphasize these activities. 
Acceptance by teachers, and to a degree principals, of this shift in the principal's role is 
by no means complete. 

Principals' employment contracts usually were silent on methods of appraisal, and 
pre-conferences, when they were held, were often tied in with reviews of the schools' goal 
packages. 

Teacher evaluation, though the most widespread and most fully implemented of all staff 
appraisal programs, nevertheless showed substantial weakness. There is a fundamental debate 
occurring at the level of organizational purposes that is reflected in confusion about the 
purposes of appraisal and the criteria to be used. Teachers express the view that they should 
be held accountable only for using appropriate teaching techniques in the classroom; 
therefore, in Lheir view, their evaluation should be limited to these and should be directed 
at developing greater personal and professional competence. In conflict with this view are 
both a holistic view that suggests teachers' contributions to the school and community are 
equally important and the view that teachers should be held accountable for the quality of the 
work done by their students. 
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The holistic view of education and the teacher's role in it are perhaps most widely 
accepted among teachers in separate schools and, especially, among francophone teachers. The 
latter group tend to accept that they have an important role to p^ay in preserving the 
vitality of French language and culture in Ontario. Anglophone teachers in public boards are 
the least likely to accept the view that their out-of-classroom activities are part of their 
job and, hence, valid areas for their appraisal. Recent debates over teachers' 
responsibilities (or lack thereof) for lunchroom and playground supervision, extracurricular 
activities, and the like illustrate the fundamental structural and role implications of this 
issue. 

The notion that teachers should be held accountable for the quality of their students' 
work has little support among professional educators at any level, but the situation may 
differ considerably among lay people. No data were collected from parents on this issue, but 
the promise of province-wide examinations by the government of Ontario suggests concern about 
results, not just methods, is present. 

Regardless of the outcome of these debates, classroo*^ instruction will remain a major 
focus of teacher evaluation and, in this area, there was a fair consensus among professional 
educators as to the purposes and criterion domains that should be used. The primary purpose 
was seen to be developmental (except, perhaps, in the case of teachers on probationary 
contracts or who are under review), and the criterion domains included classroom management, 
course and lesson organization, subject-matter knowledge, and the like. At the level of 
specific criteria and standards, however, considerable variation existed. The data suggested 
that greatest reliance was placed on the professional judgement of the evaluator rather than 
on any coherent model of effective teacher behaviour. 

In most cases, teachers' contracts were silent on the issue of appraisal, though in some 
cases detailed clauses had been developed for processes to be followed when teachers were 
placed under review and were subject to possible termination. In these cases, the purrjse of 
the appraisal became cli^arly administrative, in the sense that a decision would be made 
concerning the teacher's continued employment, and the process of planning for the evaluation 
(including pre-conferences and the like) and criteria were more thoroughly specified. 
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Data Col lection 



Data conection tended to be subordinate to criteria and potentially was affected by 
contracts, but not job descriptions and organizational structure. 

For directors, the process of collecting data might be specified in their contract, but 
in most cases these matters were detailed in public policies or guidelines adopted by the 
board. Of the two approaches, the latter is probably preferable since public documents 
announce to both the public and other staff that accountability through formal performance 
appraisal applies at the highest level. 

Superintendents generally indicated that too little information was collected and used in 
their appraisal; this concern probably reflected poorly developed and implemented appraisal 
schemes. No evidence suggested that contracts for employment dealt with this issue. 

Principals' employment contracts were usually silent on the methods by which they were to 
be appraised, hence no indication was usually present in them concerning what types, sources, 
and methods of collecting data would be used. Still, it was clear that there was a difficulty 
in the use of teacher-provided information because principals and teachers are viewed as 
co-professionals who should not make negative reports on another's performance without 
informing that individual. The data suggested that information provided by teachers was 
extremely valuable in conducting useful appraisal of principals and a significant number of 
boards were able to accommodate the principals' professional rights with the needs for a sound 
evaluation. That this was possible may in large part result from the fact principals in a 
board are generally well organized and meet regularly, thereby facilitating their involvement 
in developing appraisal policies. 

There was a clear debate, as suggested above, as to what types of data should be used in 
teacher evaluation. Where boards insisted that data be collected on out-of-classroom 
activities, this was done and teachers were appraised accordingly. As far as data on 
classroom activities were concerned, the use of information collected from students was 
opposed by most teachers; still, evidence suggested that, when used, it led to better 
appraisals. Clearly, the type of information collected from students can differ; data 
suggested that reviewing students' work and speaking to them about the class and their 
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progress can yield valid data and are not opposed by teachers to the extent that the use, say, 
•)f evaluation questionnaires completed by students would be. At present, then, the effect of 
coi^tracts on the sources and types of information used is very small, but there is a potential 
for restrictive clauses that would weaken effective appraisal methods at the school level. 

Reporting and Fol low-Up 

The most visible link between the organizational side of figure 1 and reporting and follow-uo 
were contract clauses that required notification, and perhaps involvement, of federation 
representatives if teachers (or principals) were subject to possible dismissal. In this case, 
appraisal policies usually called for thorough reporting and follow-up, including the 
provision of assistance to improve the person's work. 

Analysis suggested that post-conferences and the making of plans were important features 
of an effective appraisal system. However, there were no structural links in most cases to 
ensure that the results of one appraisal would be used in a subsequent appraisal or to assess 
the effectiveness of achieving system goals, except in the case of directors and sometimes 
superintenoents. In most cases, evaluation reports simply went on file. 

Evolution of Policy 

Formal reviews of performance appraisal policies were notable by their absence, especially for 
teachers. Policies were reviewed, it appeared, only when appeals, grievances, and lawsuits 
were filed and boards were forced to conduct reviews to prevent future problems. Principals 
seemed more able to influence their appraisal and the considerable effort put into the process 
of principal appraisal seemed to minimize the demand -- or need -- for changes. Nevertheless, 
though, there seemed to be a general agreement that the process could be improved by inclusion 
of more sources of information, including self-evaluation. Policies for the appraisal of 
superintendents and directors are still evolving. In the case of superintendents, the high 
level of discontent expressed about existing systems suggests they will try to negotiate 
changes. 

While no consensus existed among those surveyed as to the advisability of separate 
appraisal systems for developmental as opposed to administrative purposes, our own assessment 
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is that the legal problems associated with any decision to dismiss, demote, or otherwise take 
action that results in a loss to the employee virtually mandates that separate processes be 
used to ensure that valid, reliable data are collected and due process is followed. As well, 
the importance of having clear purposes for evaluation was evident in the analysis of teacher 
and principal systems. It is much easier to convey purposes clearly if they are few in number 
and do not harbour internal contradictions; whethe*^ such procedures are present in a single 
policy or two different policies is unimportant. 

Reviews of staff evaluation policies, then, are most likely to effect changes in the 
policies themselves or employment contracts. Indeed, if issues are not addressed by means of 
routine policy reviews, they will probably be addressed at the bargaining table. 

Impact of Policy and Practice 

The most important link between performance appraisal systems and the organization and its 
operations is the impact appraisal has on job performance. At best, this impact was modest at 
all levels, being highest for superintendents and principals and least for teachers and 
directors. In most cases, morale was improved and a few new ideas were conveyed; as well, 
data suggested that the very existence of an appraisal system tended to make individuals 
perform nearer to the limits of their ability than they might otherwise have done Evidence 
suggested that, while there was a relatively high level of compliance with most policies, only 
modest effort had been made to implement the policies. Greater effort, it appears, is needed 
if high levels of compliance are to become the norm. That compliance with policies was 
associated with effective evaluation programs suggests that the impact of appraisal on job 
performance can be increased. 

In some cases, evaluation clearly affected individuals' objectives. This steering effect 
was most noticeable among teachers, who were often led to place more emphasis on their overall 
performance rather than just their classroom performance. Exceptions, of course, occurred; 
some teachers reported the opposite effect. Clearly, there was a question of the proper 
balance between the two that differed in different settings and with different individuals. 
Principals, too, reported changes in personal and school objectives as the result of 
appraisal . 

29,, 

•» 276 



If performance appraisals are used to identify staff for dismissal, then staff 
associations might bargain for inclusion within the contracts of clauses that would weaken the 
use of performance appraisal for this purpose. From the public's view, such action probably 
would be considered a negative effect of performance appraisal. However, the evidence 
suggested that an effective appraisal program results in fewer dismissals and appeals, so this 
type of effect would be less likely in boards with effective systems and most likely in those 
with inadequate systems. 

Job descriptions tend not to be affected by the results of appraisal even when it is 
evident that tne appraisal system is being used, as it is in the case of principals and 
teachers, to encourage activities not fully agreed to by all parties. It would seem there is 
a need for job descriptions to ^^e altered, either within board policies, employment contracts, 
or even provincial legislation, to remove this burden from the appraisal process. 

Finally, performance appraisal at the director's and, to a degree, superintendent's level 
is directly linked, via the use of objectives, to the school board's goals and objectives. 
Yet, evidence suggested that the setting of goals at the start of the director appraisal 
process had a greater effect on the board's goals than did the results of appraisal, per se. 



Taken together, the evidence suggests that performance appraisal systems are inadequately 
linked to the organizational structures and processes within sc^ool systems. The strongest 
linkages are between organizational goals and the appraisal f directors and superintendents, 
between the principals' job descriptions and criteria for appraisal, and between principals' 
appraisal reports and their job performance. The weakest linkages are between the teachers' 
job descriptions and criteria for evaluation, and between the evaluation report for teachers 
and their job performance. 

The pattern of pre-evaluation conferencing, objective-setting, post-evaluation 
conferencing, and making of plans seems strongly related to performance appraisals that affect 
job performance ond individual objectives positively. Extension of such a pattern to all 
roles seems advisable. As well, formal assessment of appraisal systems to monitor their 
effectiveness seems logical; the current pattern of reviews in response to problems has little 



Summary 




to recommend it. Finally, attempts to define the roles of teacher and principal so that they 
align mere fully with the expectations as revealed by the criteria used in their .valuation 
are needed. As it stands, performance appraisal is being used as an instrument to bring about 
changes in practice that do not have full legitimacy in the eyes of teachers and, to some 
extent, principals. Job de'^criptions and, perhaps, contracts, laws, and regulations need to 
be changed to clarify the community's expectations for its schools and their staff. 
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